Organic Computing Doctoral Dissertation Colloquium 2018 B

13 13

S. Tomforde | B. Sick (Eds.)

Organic Computing Doctoral Dissertation Colloquium 2018 B. Sick (Eds.) Organic Computing B. | S. Tomforde S. Tomforde

ISBN 978-3-7376-0696-7

9 783737 606967

V`:%$V$VGVJ 0QJ `Q`8 `8 V`J.:`R 1H@5 J10V`1 ? :VC

Organic Computing

Doctoral Dissertation Colloquium 2018

S. Tomforde, B. Sick (Editors) !! ! "#$%&'&$'$(&)(#(&$* + "#$%&'&$'$(&)(#$&,*&+ - .! ! /)!/#0// 12#$%'$'$()(#$, 23 & ! )))0&,)(#$' '0)/#5 6 7851 999!& !

kassel university press 7 6 Preface

The Organic Computing Doctoral Dissertation Colloquium (OC-DDC) series is part of the Organic Computing initiative [9, 10] which is steered by a Special Interest Group of the German Society of Computer Science (Gesellschaft fur¨ Informatik e.V.). It provides an environment for PhD students who have their research focus within the broader Organic Computing (OC) community to discuss their PhD project and current trends in the corresponding research areas. This includes, for instance, research in the context of Autonomic Computing [7], ProActive Computing [11], Complex Adaptive Systems [8], Self-Organisation [2], and related topics. The main goal of the colloquium is to foster excellence in OC-related research by providing feedback and advice that are particularly relevant to the students’ studies and ca- reer development. Thereby, participants are involved in all stages of the colloquium organisation and the review process to gain experience. The OC-DDC 2018 took place in Wurzburg,¨ Germany on June 21st to 22nd, 2018. It was organised by the Games Engineering group at the University of Wurzburg¨ in cooperation with the Intelligent Embedded Systems group at University of Kassel. It continued the series of successful OC-DDC events: • The 2013 OC-DDC took place in Augsburg, Germany [16] • The 2014 OC-DDC took place in Kassel, Germany [17] • The 2015 OC-DDC took place in Augsburg, Germany (together with the TSOS Spring School) [18] • The 2016 OC-DDC took place in Duisburg, Germany [19] • The 2017 OC-DDC took place in Bochum, Germany [20] The 2019 edition will be part of the Informatik20191 conference, organised at Uni- versity of Kassel. Seventeen contributions have been accepted—allowing the authors to present their work in Wurzburg.¨ In addition, valuable input has been provided to the OC-DDC by Sylvain Cussat-Blanc (Universite Toulouse) and Sebastian von Mammen (Uni-

1 https://informatik2019.de/ (last access: 29/01/2019) VIII Preface versitat¨ Wurzburg)¨ as invited speakers. We would like to take this opportunity to express our gratitude to all participants and in particular to the invited speakers. This book presents the results of the OC-DDC 2018. Successful participants have been invited to extend their abstracts submitted to the event towards a full book chapter by taking reviews and feedback received at the event in Wurzburg¨ into account. The participants prepared an initial extended abstract, helped to perform a sophisticated review process, and finally came up with interesting articles summarising their current work in the context of Organic Computing. Hence, the book also gives an overview of corresponding research activities in the field in Germany for the year 2018. The collection of contributions reflects the diversity of the different aspects of Organic Computing. In the following, we outline the contributions contained in this book. Organic Computing postulates to equip technical systems with “life-like” properties. Technically, this means to move traditional design-time decisions to runtime and into the responsibility of systems themselves. As a result, systems have a dramat- ically increased decision freedom that leads to highly autonomous behaviour. The goal of this process is to allow for self-adaptation and self-improvement of system behaviour at runtime [15]. Especially since conditions that occur at runtime can only be anticipated to a certain degree, efficient mechanisms are needed that guide the system’s behaviour even in cases of missing knowledge or uncertain environmental status. Consequently, Organic Computing summarises a variety of aspects and techniques that are needed to finally develop such mechanisms [14]. For instance, a major challenge for Organic Computing and related research initiatives is the increasing in- terconnectedness of autonomous systems [13]. As a result, we face open, distributed systems consisting of autonomous elements belonging to different authorities [5, 3]. In particular, autonomous entities as part of large-scale distributed system composi- tions require secure interaction schemes [6]. In conclusion, the term Organic Computing comprises efforts to develop intelligent systems, i.e., to investigate techniques and concepts that allow for a runtime self- adaptation and self-organisation of typically distributed subsystems. Therefore, approaches from different domains are needed: distributed systems (e.g., self-organised communication, election and consensus algorithms), multi-agent systems (e.g., self- organised system organisation, negotiation), machine learning, and systems engineering. In the following, we will see that most of these fields are covered by the contributions of this year’s OC-DDC. The first group of papers is dedicated to a system perspective, although already incorporating machine learning techniques. A first contribution by Andre Bauer (University of Wurzburg)¨ deals with the question how future system states in Autonomic/Organic Computing systems can be predicted reliably. The paper is entitled “Challenges of and Approaches for Forecasting in the Context of Autonomic Computing”. Closely related in terms of the underlying techniques is the work by Markus Gorlich¨ (University of Augsburg). He focuses on predicting the next maintenance cycle, Preface IX i.e., the need for human-based exchange of components: “Predictive Maintenance in Complex Systems”. A third paper is presented by Markus Proll¨ (Hochschule Augsburg)—his work is situated in the context of industrial production and also aims at predicting system behaviour. The contribution is entitled “Transparency in Production Processes - A Big Data Analytic System Approach”. To allow for dealing with unanticipated situations and to continuously improve the system behaviour at runtime, machine learning techniques play an important role within the overall concept of an OC system [12]. In this context, Jan Schneegans and Maarten Bieshaar from University of Kassel presents their joint contribution entitled “Smart Device based Initial Movement Detection of Cyclists using Con- volutional Neuronal Networks”. They summarise the key challenges for real-world applicability of neural network-based learning techniques in the context of predicting the movements of vulnerable traffic participants such as cyclists. Transfer learning is another sub-domain in machine learning. The contribution by Simon Stieber (University of Augsburg) makes use of transfer learning concepts in the field of carbon fiber production. His work is entitled “Transfer Learning for Op- timisation of Carbon Fiber Reinforced Polymer Production”. Also in the context of transfer learning, Jens Schreiber (University of Kassel) focuses on the application area of renewable power generation. He presents the concept for his PhD thesis using the title “Transfer Learning in the Field of Renewable Ener- gies: A Transfer Learning Framework Providing Power Forecasts Throughout the Lifecycle of Wind Farms After Initial Connection to the Electrical Grid”. Wenzel Pilar von Pilchau (University of Augsburg) uses the blockchain and its various applications as target system. He analyses the applicability of machine learning algorithms to tackle challenges in this domain: “Combining Machine Learning and Blockchain: Benefits, Approaches and Challenges”. As eighth contribution, Andreas Margraf (Fraunhofer IGCV, Augsburg) makes use of a particular field in machine learning: evolutionary computation. He also works in an application-driven scenario and entitled his work with “Evolutionary Learning for Processing Pipelines in Image Segmentation and Monitoring Systems”. The Organic Computing initiative has always considered nature and biological processes as inspiration for mastering complex systems, see e.g. [4]. In this context, the next three contributions follow the path of Organic Computing. Andreas Knote (University of Wurzburg)¨ presents his work on simulating biological cell structures and the underlying biological processes efficiently. His contribution is entitled “Agent-Based Biological Cell Simulations for Self-Organising Develop- mental Processes in Morphogenesis”. The tenth paper is presented by Oliver Kosak (University of Augsburg) and entitled “Multipotent Systems: A New Paradigm for Multi-Robot Applications”. He consid- X Preface ers scenarios, where the cooperation of a potentially large set of autonomous robots is needed to solve a task. Furthermore, Adrian Calma (University of Kassel) resembles the hybrid system structure of many natural systems. He presents his work on “Decision support with Hybrid Intelligence”. The Organic Computing initiative has its roots in the technical computer science domain [1]. Since the very beginning of the initiative, questions of how to realise self-adaptation and self-learning behaviour in computer architecture have been part of the major focus. This year, we have three contributions that continue research in this direction. Norbert Schmitt (University of Wurzburg)¨ focuses on server infrastructure and the corresponding efforts to allow for self-administration. His work is entitled “Energy Efficiency of Server, Systems and Software”. Next, Armin Sadighi (TU Munich) investigates self-adaptive multi-processor architectures. His contribution is entitled “Self-Aware MPSoC Architecture Extensions for Automotive Applications”. Michael Bromberger (Karlsruhe Institute of Technology) considers adaptivity in the context of cross-layer optimisation. His paper “Tuning the Computational Effort: An Adaptive Accuracy-aware Approach Across System Layers” proposes computational efficient concepts. Finally, a major goal of Organic Computing is to transfer the developed techniques and methods from research into real-world applications. Three of the contributions of this book have a strong focus on such an application—although coming from very heterogeneous fields. The work by Tim Wachter¨ (Fachhochschule Bielefeld- Minden) is entitled “Dynamical Evacuation Route System for Intelligent Buildings” and proposes a concept for autonomously deriving and indicating rescue routes in large buildings using reconfigurable hardware. Ferdinand von Tullenburg¨ (Salzburg Research Forschungsgesellschaft) investigates the mutual influences between communication infrastructure and future smart grid systems. His work is entitled “Organic-Computing inspired Robustness Assessment of Smart Grids”. The focus of the work presented by Lena Elisa Schneegans (University of Kassel) is on smart cities, i.e., on the intelligent control of inner-city traffic. Her contribution is presented under the title “Multi-Agent Model Predictive Control in Urban Traffic Networks”. We thank all authors for their contributions and we are looking forward to seeing again very interesting OC research at the next OC-DDC.

Kassel, Sven Tomforde February 2019 Bernhard Sick Preface XI References

1. Beigl, M., Brinkschulte, U., Feldbusch, F., Fey, D., Fischer, S., Hochberger, C., Hoff- mann, R., Karl, W., Krebs, J., Kleinoder,¨ J., Lagally, K., Langhammer, F., Lukowicz, P., Marwedel, P., Maehle, E., Muller-Schloer,¨ C., Schallenberger, B., Schmeck, H., Tavan- garian, D., Trumler, W., Ungerer, T., Waldschmidt, K.: Grand challenges der technischen informatik. Report, Gesellschaft fur¨ Informatik eV (2008), https://www.informatik.uni- augsburg.de/de/lehrstuehle/sik/downloads/GC-Report-Maerz08.pdf 2. Camazine, S., Deneubourg, J.L., Franks, N.R., James Sneyd and, G.T., Bonabeau, E.: Self-Organization in Biological Systems. Princeton Univers. Press (2003) 3. Diaconescu, A., Frey, S., Muller-Schloer,¨ C., Pitt, J., Tomforde, S.: Goal-oriented Holonics for Complex System (Self-)Integration: Concepts and Case Studies. In: Proceedings of the 10th IEEE International Conference on Self-Adaptive and Self- Organising Systems, held September 12-16, 2016 in Augsburg, Germany. pp. 100 – 109. IEEE (2016) 4. Edenhofer, S., Tomforde, S., Fischer, D., Hahner,¨ J., Menzel, F., von Mammen, S.: De- centralised Trust-management Inspired by Ant Pheromones. International Journal of Mobile Network Design and Innovation, special Issue on “Signal Processing, Security and Privacy for Mobile/Wireless and Computer Networks” p. to appear (2016) 5. Hahner,¨ J., Brinkschulte, U., Lukowicz, P., Mostaghim, S., Sick, B., Tomforde, S.: Run- time Self-Integration as Key Challenge for Mastering Interwoven Systems. In: Proceed- ings of the 28th GI/ITG International Conference on Architecture of Computing Systems – ARCS Workshops, held 24 – 27 March 2015 in Porto, Portugal, Workshop on Self- Optimisation in Organic and Autonomic Computing Systems (SAOS15). pp.1–8.VDE Verlag (2015) 6. Hahner,¨ J., Rudolph, S., Tomforde, S., Fisch, D., Sick, B., Kopal, N., Wacker, A.: A Concept for Securing Cyber-Physical Systems with Organic Computing Techniques. In: Workshop Proceedings of the 2013 International Conference on Architectures of Com- puting Systems (ARCS’13) – Workshop on Self-optimisation in Organic and Autonomic Computing Systems (SAOS’13), held February 19th - 22nd, 2013, Prague, Czech Re- public. VDE Verlag (2013), http://www.vde-verlag.de/proceedings-de/563492001.html 7. Kephart, J.O., Chess, D.M.: The Vision of Autonomic Computing. IEEE Computer 36(1), 41–50 (2003) 8. Kernbach, S., Schmickl, T., Timmis, J.: Collective adaptive systems: Challenges beyond evolvability. ACM Computing Research Repository (CoRR) (2011), ftp: //ftp.cordis.europa.eu/pub/fp7/ict/docs/fet-proactive/shapefetip-cas09\ en.pdf, last access: 14/07/2014 9. Muller-Schloer,¨ C., Schmeck, H., Ungerer, T. (eds.): Organic Computing - A Paradigm Shift for Complex Systems. Autonomic Systems, Birkhauser¨ Verlag (2011) 10. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 11. Tennenhouse, D.: Proactive computing. Communications of the ACM 43(5), 43–50 (2000) 12. Tomforde, S., Brameshuber, A., Hahner,¨ J., Muller-Schloer,¨ C.: Restricted On-line Learning in Real-world Systems. In: Proc. of the IEEE Congress on Evolutionary Com- putation (CEC11), held 05 Jun - 08 Jun 2011 in New Orleans, USA. pp. 1628 – 1635. IEEE (2011) XII Preface

13. Tomforde, S., Hahner,¨ J., Sick, B.: Interwoven Systems. Informatik-Spektrum 37(5), 483–487 (2014), http://dx.doi.org/10.1007/s00287-014-0827-z, Aktuelles Schlagwort 14. Tomforde, S., Muller-Schloer,¨ C.: Incremental Design of Adaptive Systems. Journal of Ambient Intelligence and Smart Environments 6, 179 – 198 (2014) 15. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 16. Tomforde, S.: Computing doctoral dissertation colloquium (oc-ddc?13). Tech. rep., Uni- versity of Augsburg, Faculty for Applied Computer Science, Tech. Rep. TR 2013-06 (2013) 17. Tomforde, S., Sick, B.: Organic Computing: Doctoral Dissertation Colloquium 2014 (2014) 18. Tomforde, S., Sick, B.: Organic Computing: Doctoral Dissertation Colloquium 2015 (2015) 19. Tomforde, S., Sick, B.: Organic Computing: Doctoral Dissertation Colloquium 2016 (2016) 20. Tomforde, S., Sick, B.: Organic Computing: Doctoral Dissertation Colloquium 2017 (2017) Invited Keynote

Introduction to Gene Regulatory Networks

Abstract. Gene regulatory networks are a central mechanism in the regulation of gene expression in all living organisms cells. Their functioning is nowadays very well understood: they are based on the production of proteins enhanced or inhibited by other proteins from the inside and/or the outside of the cells. This produces complex dynamics with which cells from the same organism, with the exact same DNA, can form very different types (through cell differentiation) and have very different behaviours according to their types and positions. This talk will first introduce the biology of these networks, from the genetic aspect to the dynamics of gene regulation. Then, biologically plausible models will be presented to highlight the complexity of dynamics gene regulatory networks can produce. Using that model, we will show how computational models can be designed so that a genetic algorithm can optimise the network efficiently. We will present some applicative examples in which artificial gene regulatory networks are plugged into diverse virtual agents (artificial cells in an artificial embryogenesis context, direct connection to the sensors and effectors or high-level behaviour regulation in others).

Sylvain Cussat-Blanc Institute of Research in Informatics of Toulouse, University of Toulouse XIV Invited Keynote Hands-On Complex Systems Research - Interactive Self-Organisation

Abstract. We have taken numerous paths and developed plenty of methods to understand and master complex systems. Yet, no matter how great the degree of autonomy of the corresponding technological solutions, they are always embedded into a human context. Interactive Self-Organisation gives credit to this insight and suggests that at various levels of abstraction (from the individual to subsets of individuals to the whole), one or more users should be given the means to inspect and inﬂuence these technical solutions. In this keynote presentation, this topic is motivated based on various application domains and numerous examples of Interactive Self- Organisation are given.

Sebastian von Mammen Human-Computer Interaction, University of Wurzburg¨ Contents

Part I Mastering Complex Systems

1 Challenges and Approaches: Forecasting for Autonomic Computing Andre´ Bauer ...... 3 2 Predictive Maintenance in Complex Systems Markus Gorlich¨ ...... 21 3 Towards an Agent-based Big Data Analytics Microservice System Sebastian Proll¨ ...... 29

Part II Machine Learning as Basis for Organic Computing

4 Smart Device Based Initial Movement Detection of Cyclists Using Convolutional Neural Networks Jan Schneegans and Maarten Bieshaar ...... 45 5 Transfer Learning for Optimisation of Carbon Fiber Reinforced Polymer Production Simon Stieber ...... 61 6 Transfer Learning in the Field of Renewable Energies Jens Schreiber ...... 75 7 Combining Machine Learning with Blockchain: Beneﬁts, Approaches and Challenges Wenzel Pilar von Pilchau ...... 89

8 Evolutionary Learning for Processing Pipelines in Image Segmentation and Monitoring Systems Andreas Margraf ...... 99 XVI Contents

Part III Nature and Social Systems as Inspiration for Organic Computing

9 Interactive Agent-Based Biological Cell Simulations for Morphogenesis Andreas Knote and Sebastian von Mammen ...... 115 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications Oliver Kosak ...... 125 11 Decision Support with Hybrid Intelligence Adrian Calma and Dominik Dellermann ...... 143

Part IV Self-adaptation and Self-learning in Computer Architecture

12 Improving the Energy Efﬁciency of IoT-Systems and its Software Norbert Schmitt ...... 157 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications Armin Sadighi, Thomas Wild, and Andreas Herkersdorf ...... 173 14 Tuning the Computational Effort: An Adaptive Accuracy-aware Approach Across System Layers Michael Bromberger ...... 185

Part V Organic Computing in Real-world Applications

15 Dynamical Evacuation Route System Tim Wachter¨ ...... 197 16 Robustness Modeling and Assessment of Interdependent Smart Grids Ferdinand von Tullenburg¨ ...... 211 17 Survey on Urbantrafﬁc Control and Management Lena Elisa Schneegans ...... 223 List of Contributors

Andre´ Bauer Simon Stieber Chair of Software Engineering Institute for Software & Systems Engineer- University of Wurzburg¨ ing Am Hubland University of Augsburg 97074 Wurzburg,¨ Germany Universitatsstraße¨ 6a [email protected] 86135 Augsburg, Germany [email protected]

Markus Gorlich¨ Jens Schreiber Organic Computing Group Intelligent Embedded Systems Lab University of Augsburg University of Kassel Eichleitnerstraße 30 Wilhelmshoher¨ Allee 73 86159 Augsburg, Germany 34121 Kassel, Germany [email protected] [email protected]

Sebastian Proll¨ Wenzel Pilar von Pilchau Distributed Systems Group Organic Computing Group University of Augsburg University of Augsburg An der Hochschule 1 Eichleitnerstraße 30 86161 Augsburg, Germany 86159 Augsburg, Germany [email protected] Wenzel.Pilar-von-Pilchau@informatik. uni-augsburg.de

Jan Schneegans Andreas Margraf Intelligent Embedded Systems Lab Online Process Monitoring University of Kassel Fraunhofer IGCV Wilhelmshoher¨ Allee 73 Am Technologiezentrum 2 34121 Kassel, Germany 86159 Augsburg, Germany [email protected] [email protected]

Maarten Bieshaar Andreas Knote Intelligent Embedded Systems Lab Human-Computer Interaction University of Kassel University of Wurzburg¨ Wilhelmshoher¨ Allee 73 Am Hubland 34121 Kassel, Germany 97074 Wurzburg,¨ Germany [email protected] [email protected] XVIII List of Contributors

Sebastian von Mammen Prof. Dr. Andreas Herkersdorf Human-Computer Interaction Chair of Integrated Systems University of Wurzburg¨ Technische Universitat¨ Munchen¨ Am Hubland Arcisstraße 21 97074 Wurzburg,¨ Germany 80290 Munchen,¨ Germany [email protected] [email protected]

Oliver Kosak Michael Bromberger Institute for Software & Systems Engineer- Computer Architecture and Parallel ing Processing University of Augsburg Karlsruhe Institute of Technology Universitatsstraße¨ 6a Kaiserstraße 12 86135 Augsburg, Germany 76131 Karlsruhe, Germany [email protected] [email protected]

Adrian Calma Tim Wachter¨ Intelligent Embedded Systems Lab Institute of the Intelligent Building University of Kassel University of Applied Science Bielefeld, Wilhelmshoher¨ Allee 73 Campus Minden 34121 Kassel, Germany Artilleriestraße 9 [email protected] 32427 Karlsruhe, Minden [email protected]

Dominik Dellermann Ferdinand von Tullenburg¨ vencortex Salzburg Research Forschungsgesellschaft Kassel, Germany mbH [email protected] Jakob-Haringer-Straße 5/3 5020 Salzburg, Austria [email protected]

Norbert Schmitt Lena Elisa Schneegans Chair of Software Engineering Trafﬁc Engineering and Transport Logistics University of Wurzburg¨ University of Kassel Am Hubland Monchebergstrae¨ 7 97074 Wurzburg,¨ Germany 34125 Kassel, Germany [email protected] [email protected]

Armin Sadighi Prof. Dr. Sylvain Cussat-Blanc Chair of Integrated Systems Institute of Research in Informatics of Technische Universitat¨ Munchen¨ Toulouse Arcisstraße 21 University of Toulouse 80290 Munchen,¨ Germany Avenue de l’tudiant [email protected] 31400 Toulouse, France [email protected] Dr. Thomas Wild Chair of Integrated Systems Technische Universitat¨ Munchen¨ Arcisstraße 21 80290 Munchen,¨ Germany [email protected] List of Contributors XIX

PD Dr. Sven Tomforde Prof. Dr. Bernhard Sick Intelligent Embedded Systems Lab Intelligent Embedded Systems Lab University of Kassel University of Kassel Wilhelmshoher¨ Allee 73 Wilhelmshoher¨ Allee 73 34121 Kassel, Germany 34121 Kassel, Germany [email protected] [email protected]

Part I

Mastering Complex Systems

Challenges and Approaches: Forecasting for Autonomic Computing

Andre´ Bauer

Chair of Software Engineering, University of Wurzburg,¨ Wurzburg,¨ Germany [email protected], Home Page: https://descartes.tools

Abstract. In a fast-paced world, software systems require autonomic management. In order to enable accurate and proactive autonomic systems, reliable time series forecasting methods are required. However, most forecasting methods have either a high variance of accuracy and/or time-to-result. To this end, forecasting methods with robust performance are demanded. In order to select the best-suited approach, a fair benchmark of forecasting methods is needed. In this work, the challenges of forecasting in autonomic computing and the associated approaches are outlined.

Keywords: Forecasting, Machine Learning, Benchmark

1.1 Introduction

Nowadays, software systems are pushed to their limits, on the one hand, by the fast living and changing requirements of their users and, on the other hand, by a huge amount of data that they create or have to process. Although cloud computing is a paradigm that allows facing the increasing scale and complexity of modern software, clouds respectively data centers require skilled IT professionals for configuration and maintaining these systems. Already in 2001, IBM released the vision of autonomic computing as they observe that programmes or environments tend to grow more and more complex. Further, they assume that this complexity exceeds the human capacity [17]. Almost 20 years later, we have billions of interconnected devices (smartphones, IoT, ...), an innu- merable amount of services/applications, and approaches such as organic computing [22, 27], self-aware computing [19], and more that build upon this vision. All these approaches have in common that they are equipped with sensors and interact with their environment [26]. Based on the observations, the systems plan and 4 A. Bauer execute actions either to adjust to the environment or adapt the environment. How- ever, when triggering actions based on observations, the system only reacts. These actions have an inherent delay that may lead to problems. In nature, animals or humans have the ability, which is commonly called intuition, to ”predict” upcoming events. In order to upgrade the systems with this ability, i.e., allowing the system to be proactive, the systems require accurate and reliable forecasting methods. In many research areas, forecasting is an established but important discipline that allows predicting the future by examining the past observations. Based on the ”No- Free-Lunch Theorem” [29] of 1997, which states that there is no optimisation algorithm that is best suited for all scenarios, an analogy can be drawn to the forecasting of time series since the methods try to model the historical data best. So there is no forecasting method that performs best for all time series. In other words, forecasting methods have their benefits and drawbacks depending on the specific use cases respectively time series. Indeed, expert knowledge is required for choosing the best forecasting method. However, expert knowledge is expensive, may take a long time to deliver results, and it cannot be completely automated. In order to overcome this, this work identifies problems and the corresponding 7 challenges of forecasting in the context of autonomic computing. These challenges are grouped in online, offline, and benchmarking challenges. Further, this work proposes approaches to face these challenges, suggests auto-scaling as a use case of a proactive automatic system, and shows some preliminary results. The remainder of this work is structured as follows: The next section presents briefly foundations regarding time series and forecasting methods. In Section 1.3, the challenges of forecasting in the context of autonomic computing are stated. Section 1.4 presents the ideas to face the challenges and suggest an use case for an proactive autonomic system. Preliminary results of tackling one of the challenges are discussed in the Section 1.5. In Section 1.6, related work is summarised before the paper is concluded.

1.2 Time Series Forecasting

In order to better understand the following sections respectively recapitulate common terms, this section introduces foundations from the ﬁeld of time series and forecasting methods.

1.2.1 Time Series Foundations

This section gives a short introduction to common terms in the context of forecasting. A univariate time series is a sequence of data points ordered by equidistant time steps. 1 Challenges and Approaches: Forecasting for Autonomic Computing 5

Mathematically, let yt ∈ R be an observation at date t, the univariate time series is deﬁned by { }T . Y: yt t=1 Especially in the context of autonomic computing, systems have several sensors. Either the observations of each sensor can be saved as a univariate time series or the observations of correlated sensors can be stored as a multivariate time series. In other words, a time series can also be multivariate and has for each date at least two observations. In the following, the term time series is used for a univariate time series.

1.2.1.1 Decomposition

A further mathematical deﬁnition of a time series can be derived by decomposing the time series into its components. Indeed, there are different decomposition methods that lead to diverse components. In this work, STL (Seasonal and Trend decomposition using Loess) [9] is considered for time series decomposition. STL is a commonly used method, where the time series is decomposed in the trend component T, the seasonal component S, and the irregular component I or noise:

Y(t)=T(t) S(t) I(t).

The operation depends on whether the time series has an additive or multiplicative decomposition. The trend is the long-term movement in time series, i.e., upwards, downwards, or stagnate. Usually, the trend is a monotone function unless extrinsic events trigger a break and cause a change in the direction. The presence of recurring patterns within a regular period in time series is called seasonality. These patterns are caused by climate, customs, or traditional habits such as night and day phases. Figure 1.1 depicts an exemplary decomposition of a time series.

1.2.1.2 Box-Cox Transformation

As observed data may contain complex patterns, an adjustment respectively sim- pliﬁcation of this data can improve the forecasting model. To this end, there exist different methods. A useful approach is the Box-Cox transformation [5]. This transformation depends on the transformation parameter λ. The parameter can be estimated with the method introduced by Guerrero [12]. The Box-Cox transformation is deﬁned as follows: ln(y ) if λ = 0; w = t t ( λ − )/λ yt 1 otherwise. 6 A. Bauer

Orginal Time Series 600

400

200 1952 1956 1960 Trend Component 500 400 300 200 100 1952 1956 1960 Seasonal Component

40 0 −40 1952 1956 1960 Irregular Component

−50 1952 1956 1960

Fig. 1.1: Examplary time series decomposition.

Based on this model, the forecast is conducted. Then, the forecast is re-transformed with the same λ: ( ) λ = = exp wt 0; yt 1/λ (λwt + 1) otherwise.

1.2.1.3 Error Measurements

In order to evaluate the accuracy of forecasting results, several error measurements can be used. Each method has its use cases, beneﬁts, and drawbacks. Thus, the selection of the measurements has to be done carefully. A common method for error measurement is the mean absolute percentage error:

1 T y − f MAPE = ∑| t t |. T t=1 yt

In this equation, ft is the forecast value, yt the observation at date t, and T the observed period of time. The MAPE is chosen for reﬂecting the forecast accuracy due to its measure characteristics: (i) It represents the percentage of average absolute error compared to the original observations. (ii) Also, the MAPE is independent of the scale of the measurement. (iii) Further, due to the absolute error, contrary signed errors do not offset each other. [1] 1 Challenges and Approaches: Forecasting for Autonomic Computing 7

1.2.2 Forecasting Methods

As forecasting is an important aspect in many research fields, there exist different methods from different disciplines. In this work, three methods are outlined as they are part of the approach: (i) an artificial neuronal network method from the field of deep learning, (ii) seasonal ARIMA from the field of statistical methods, and (iii) extreme gradient boosting from the field of machine learning.

1.2.2.1 ANN

The Artiﬁcial Neuronal Network is a feed-forward neuronal network that is trained with lagged values of a time series. In contrast to other neuronal networks, the feed- forward neural network moves the information only in one direction. That is, there are no loops or cycles in the network. In order to forecast time series, this network consists of one hidden layer. In addition, the number of lags and nodes in the hidden layer are automatically selected. [15]

1.2.2.2 sARIMA

In 1938, H. Wold lays the groundwork for using ARMA models for time series. An ARMA model is a combination of an autoregressive AR(p) model and a moving- average MA(q) model. Here, the order p determines the number of past values and the order q determines the number of past errors. As ARMA models require a stationary time series, ARIMA models overcome this restriction with differencing of the time series. In contrast to ARIMA, sARIMA is capable of modeling seasonal data. To this end, each non-seasonal component of the ARIMA model is extended with its seasonal counterpart. [16]

1.2.2.3 XGBoost

In 2014, a scalable end-to-end tree boosting system called eXtreme Gradient Boost- ing was released. This method uses tree boosting. That is, a tree ensemble model with additive functions are created. Each function corresponds to an independent tree structure. In terms of time series forecasting, the leaves of regression trees are summed up to predict the output. The method tries to select a model with simple and predictive functions. This is done in an additive, greedily manner that is also known as gradient tree boosting. [6] 8 A. Bauer 1.3 Challenges

In this section, the challenges, which arise in the ﬁeld of autonomic/organic computing, are outlined. The following challenges are grouped into three groups: (i) general challenges, (ii) online challenges, and (iii) benchmark challenges. The general challenges occur before applying time series forecasting. The online challenges arise in addition to the general challenges in an online scenario. The benchmark challenges concern themselves with how to compare different forecasting methods.

1.3.1 General Challenges

Based on its former work [30], D. Wolpert postulated the ”No-Free-Lunch Theorem” [29] in 1997. It claims that there is no optimisation algorithm that performs best for all scenarios. That is, improving the performance of one aspect normally leads to a degradation in performance for some other aspect. As forecasting methods try to best ﬁt to historical data, parallels can be drawn to optimisation problems. In other words, if a forecasting method is tuned for a speciﬁc time series, it gets worse for another time series. Thus, there is no forecasting method that outperforms the others for all time series. In order to underline this statement, a measurement over 50 time series is conducted. The forecast error and time-to-result are listed in Table 1.1. For instance, The forecasting method ANN performs worst according to the accuracy and has a standard deviation of more than 300%. In contrast, sARIMA has the lowest average forecast error, but has the highest average time-to-result and also the highest standard deviation. Based on this example and the theorem, the following challenge can be formulated: Challenge 1

How to build/ﬁnd a generic forecasting approach that delivers robust forecasting accuracy?

Here, robust means that the variance in forecasting results should be reduced, not necessarily improving the forecasting accuracy itself.

Table 1.1: Accuracy and time-to-result comparison.

MAPE Time-to-Result Method Avg [%] SD [%] Avg. [s] SD [s] ANN 80.98 306.56 31.75 91.88 sARIMA 24.92 41.14 82.32 248.52 XGBoost 49.43 162.39 0.011 0.01 1 Challenges and Approaches: Forecasting for Autonomic Computing 9

As autonomic systems have multiple sensors, software systems tend to generate a high amount of different and heterogeneous data. This results in multivariate time series. Although there is an extension for ARIMA for handling multivariate time series, the most statistical methods do not support multivariate time series. Thus, machine learning algorithms are used as forecasting tools. Due to the nature of these data, there is a need for data selection. This is also known as feature selection in the context of machine learning. As the selection is a crucial part for the quality, the following challenge can be formulated:

Challenge 2

How to automatically select the important features for forecasting?

Besides the feature selection, also the transformation of the data inﬂuence the forecasting accuracy. Hence, transforming the historical data may lead to simpler patterns that usually allows more accurate forecasts [16]. There are different kinds of transformations as calendar adjustments or mathematical transformations. As calendar adjustments require human intervention, only mathematical transformations are suitable in autonomic systems. Thus, the subsequent challenge can be posed:

Challenge 3

How to automatically transform the features for increasing the forecast accuracy?

This challenge is tackled in a preliminary evaluation in Section 1.5.

1.3.2 Online Challenges

While forecasting, for example, the visitors of a zoo has no time constraints, the time-to-result in the context of autonomic computing has strict requirements. How- ever, time to result varies heavily for traditional approaches as shown in Figure 1.2 or in Table 1.1. The diagram shows the time-to-results distribution of the introduced methods in Section 1.2.2. Each method forecasts 50 time series and is depicted on the horizontal axis. The vertical axis is in log scale and shows the time-to-result in milliseconds. For instance, the time-to-result of ARIMA ranges from approximately 200 milliseconds to 2,000,000 milliseconds. Similarly, ANN and XGBoost show both a high degree of variance. The table lists forecast error and time-to-result. For instance, sARIMA has a standard deviation of 248.52% for the time-to-result. Therefore, the next challenge can be stated: 10 A. Bauer

Challenge 4

How to build/ﬁnd a generic forecasting approach that has a reliable time-to-result?

Here, reliable means that the variance in time-to-result should be reduced without degrading the time-to-result itself.

Time Distribution

1e+06

1e+04 Time [ms]

1e+02

ANN sARIMA XGBoost Methods

Fig. 1.2: Examples for Challenge 4.

As autonomic systems may plan actions in advance, the future values have to be forecasted. The longer the forecast, the less accurate is the forecast towards the end. Thus, the planned actions may be updated as new data may influence the decision process. In other words, the regular incoming information allows quantifying the accuracy of the forecast. In order to update the forecasts, there are different approaches. One solution is to build a new forecast model with each new observation. Another option is to refine the model with the new data if this is possible for the forecasting method. In contrast to building a new model or updating the model with the incoming data, a further approach is to use the old model until a significant deviation between observation and forecast take place. Indeed, each of this approach has its strengths and weaknesses. That is, there is a trade-off between time-to-result and the accuracy of the forecast. In order to investigate this trade-off, the following challenge can be formulated: 1 Challenges and Approaches: Forecasting for Autonomic Computing 11

Challenge 5

How to reﬁne/reﬁt the forecast model with incoming data?

1.3.3 Benchmarking Challenges

As forecasting is an important part of the decision-making process and is used in many ﬁelds, there is a broad range of academic work concerning forecasting. How- ever, the degree of quality of the evaluations suffers, on the one hand, due to the lack of commonly used respectively good data sets or, on the other hand, on the methodology. For instance, the proposed forecasting methods (see Section 1.6) only predict the next value. From a statistical perspective, only the next value is important, but planning in advance requires multiple values. However, there is one work out of the surveyed ones that forecasts up to 15 points. Indeed, this is also a short range in the context of automatic computing, where a ﬁne granularity leads to several data points in a short time span. Further, the surveyed papers compare their approaches on a small data set ranges from 1 to 10 time series. Also, a comparison with other methods either is missing or contains only a few methods. Upon this, the authors only consider the forecast accuracy. By taking these limitations into account, the next challenge can be stated:

Challenge 6

How to compare different forecasting methods in a fair manner?

As stated in Section 1.2.1, there exists several measures for evaluating the accuracy of a forecast. Although the measure MAPE is used for this work based on its advantages, the selected measure has some characteristics that introduce issues in the context of automatic computing. For example, suppose there are two different time series, each with a forecast. The two forecast values f1 = 3, f2 = 3 are predicted for the observations y1 = 2, y2 = 4. In other words, the ﬁrst forecast overestimates and the second forecast underestimates the observation. Although both forecast values differ by one, the error measure has two different values: MAPE1 = 0.5 and MAPE2 = 0.25. That is, MAPE prefers forecasting methods that underestimate the observations. In practice, underestimation may lead to faults. Thus, additional metrics or new metrics have to be considered while comparing forecasting methods. The resulting challenge is:

Challenge 7

What are suitable/reliable metrics for quantifying the forecast? 12 A. Bauer 1.4 Approaches and Ideas

This section presents the ideas to face the identiﬁed challenges from Section 1.3. On the one hand, a hybrid forecasting method is envisioned and, on the other hand, a benchmark for forecasting method is planned. Further, a use case showing the importance of forecasting in the context of autonomic computing is outlined.

1.4.1 Hybrid Forecasting

In order to tackle Challenge 1 and 4, the idea is to design a hybrid machine learning mechanism that combines different forecasting methods. The preliminary version is able to improve the forecast accuracy and to reduce the variance of the results while maintaining a short time-to-result on two time series [32]. The proposed hybrid forecasting approach is called Telescope according to the analogy with the vision of far-distanced objects. The mechanism is implemented in R and can be found at GitHub1. The current version supports only univariate time series with a seasonal pattern. The core idea of the approach is to leverage additional information from the time series itself and consists of three fundamental steps: (i) Preprocessing of the time series, (ii) building the forecast model, and (iii) forecasting the future behaviour of the time series. In the preprocessing step, the frequency, i.e., the length of a period, of the time series is estimated and anomalies are removed. In this approach, the assumption is that the frequencies match the human behaviour (e.g., hourly, daily, weekly, ...). For the model learning, the machine learning algorithm eXtreme Gradient Boosting is chosen due to its success at Kaggle challenges2. In order to create features for XGBoost, the time series is decomposed via STL in its components: trend, season, and noise. Further, a second season is determined from the time series. This second frequency allows capturing a better seasonal model as, for example, a time series has a weekly pattern (ﬁrst season retrieved by STL) and a daily pattern. With the usage of clustering and ANN, this second seasonal pattern is detected. Afterward, the model is trained with both seasonsonal patterns and the trend. The irregular component is ignored since it is hard to predict and so, correlated with a high error rate. For the forecasting, the seasonsal patterns and the trend have to be forecast. As the seasonsonal patterns are per deﬁnition recurring, both patterns can be pursued. In contrast, the trend is predicted with ARIMA. Finally, XGBoost combines the ”predicted” components to forecast the time series. To face Challenge 2 and to enhance Telescope for multivariante time series, an autonomic feature selection process is envisioned. As the current version uses XGBoost

1 https://github.com/DescartesResearch/telescope (last access: 11/10/2018) 2 https://github.com/dmlc/xgboost/tree/master/demo (last access: 11/10/2018) 1 Challenges and Approaches: Forecasting for Autonomic Computing 13 as machine learning approach, a further improvement is to dynamically choose the most suitable machine learning approach based on the time series characteristics. Also, the extension to support multivariate time series is intended. In order to tackle Challenge 5 and improve Telescope for online usage, different strategies of model reﬁnement will be investigated.

1.4.2 Forecast Benchmark

Due to the lack of good data sets and the sometimes arbitrary evaluations (see Chal- lenge 6), the idea is to establish a forecast benchmark. On the one hand, the concept is that the benchmark takes control over the evaluation. This leads to a uniform evaluation of forecasting methods. On the other hand, the benchmark provides a broad data set with a high degree of diversity. Further, the data set is split into different use cases, for instance, time series that are related to the finance sector. This allows benchmarking forecasting methods in different fields. Indeed, there are data sets such as the M3 Competition3 that contains 3003 time series from different domains. However, most time series have a high degree of similarity and a length below 100 data points. In other words, this data set is, for instance, not suitable for validating forecasting methods in the context of autonomic computing as time series in this domain are generally larger. For instance, when sampling data each second, actions that are planed hourly have to take 3600 data points into account. Besides the data set, fair competition among the forecasting methods is required. That is, the forecast benchmark has to fulfil the requirements of benchmarking principles. As highlighted in Challenge 7, the current metrics may not be capable of comparing different forecasting methods in a fair manner. Thus, the benchmark has to introduce also a new set of metrics. Besides, the benchmark requires a good reporting system and usability.

1.4.3 Use Case

Autonomic systems are the solution for problems that are to complex for human beings. In the course of digitisation, the Internet usage is fast-paced and has changing requirements. Also, the increasing amount of connected devices has an inﬂuence on the usage. In the last years, cloud computing emerged as a computing model that allows fast access to resources and has a high level of scalability. Due to these beneﬁts and the requirement of the Internet usage, the deployment of autonomic resource management systems arose in cloud computing. In practice such

3 https://forecasters.org/resources/time-series-data/m3-competition/ (last access: 11/10/2018) 14 A. Bauer as in AWS (Amazon Web Services), reactive mechanisms are used that trigger releas- ing respectively provisioning of resources. As the name indicates, these mechanisms react to workload changes and try to adapt the resources to the current workload. However, these approaches are pragmatic and the actions are delayed. The lag can be seen in Figure 1.3. This diagram shows the scaling behaviour of the reactive auto- scaler introduced by Chieu et al. [7]. On the horizontal axis the time in minutes and on the vertical axis the amount of resources are depicted. The blue dashed curve depicts the demanded amount of resources. The black curve shows the supplied resources by the auto-scaler. Especially when the demand changes, the adaptions of the auto-scaler lags behind. Thus, the idea is to extend such reactive methods to be proactive respectively improve the forecast accuracy of already proactive auto-scalers in order to achieve a better auto-scaling performance. Here, the performance is measured by metrics proposed by SPEC4 [14]. In context of the use case, one work was published concerning about the input of auto-scalers [3] and another work introduces a new proactive mechanism with a reactive fallback [4].

Scaling Behavior of React 15 Demanded Resources Supplied Resources

5 Amount of Resources

0 0 50 100 150 Time in minutes

Fig. 1.3: Example illustration for the scaling behaviour of React.

1.5 Preliminary Results: Feature Engineering

In order to investigate the inﬂuence of feature transformation and tackle Challenge 3, preliminary experiments are conducted. To this end, the forecast accuracy of ANN and sARIMA are observed while transforming the historical data. The used data set contains the fma package from R. For the experiments four transformations are used: (i) identity√ function Id: x → x, (ii) natural logarithm Log: x → ln(x), (iii) square root Sqrt: x → x, and (iv) box-cox transformation Box (see Section 1.2.1.2). Table 1.2 shows for both ANN and sARIMA the proportion of the achieved rank while using a certain transformation. For instance, sARIMA with Log achieves on 36% of the time series the best forecast compared to the other transformations. In

4 Standard Performance Evaluation Corporation 1 Challenges and Approaches: Forecasting for Autonomic Computing 15

Table 1.2: Feature transformations and the resulting forecast performance.

ANN sARIMA Rank Id Log Sqrt Box Id Log Sqrt Box 1 34% 32% 7% 31% 31% 36% 10% 29% 2 8% 27% 43% 24% 11% 22% 45% 20% 3 25% 23% 35% 19% 20% 19% 42% 23% 4 33% 18% 14% 25% 37% 23% 04% 28% other words, when choosing Id for ANN, there is the probability of 34% that this transformation is the best one compared to the other ones. However, in 33% Id leads to the worst forecast. For both forecasting methods, Sqrt achieves in less than 11% the best forecast. Further, it can be stated that adjusting the data in these experiments (in ANN 64% and sARIMA 69% cases of the time series) increase the forecast accuracy. To sum up, there is no transformation that outperforms the other ones and thus, the selection is crucial. Based on the information from Table 1.2, different transformation strategies are investigated. First, each strategy adjusts with a speciﬁc transformation the data. Then, the adjusted time series is forecast by both ANN and sARIMA. Afterward, the mean rank of the forecast of ANN and sARIMA is reported. The observed strategies are: (i-iv) using the associated transformation on each time series, (v) choosing a random transformation (Random), and (vi) the proposed automatic approach for the transformation selection (Aut). The latter approach adjusts each time series with each of the four transformations. Then, for each transformation the forecast model is built. Based on the model accuracy, the best transformation is selected. Afterward, the chosen adjusted time series is forecast. Finally, for each strategy the forecast is re-transformed and the accuracy is calculated.

Table 1.3: Average performance of different strategies.

Id Log Sqrt Box Random Aut Avg. Rank 2.60 2.14 2.48 2.44 2.50 2.42

The average ranks over all time series and both forecasting methods of the different approaches are depicted in Table 1.3. Due to the 4 transformations, the ranks range from 1 to 4. Random achieves an average rank of 2.5. That is, when choosing a transformation randomly, the forecast is either the second or third best forecast. The best rank is achieved while transforming the data with Log. The automatic selection of the transformation has a rank of 2.6. That is, it is better to choose random selection than this approach. 16 A. Bauer

Table 1.4 investigates why Aut has the worst rank. Therefore, for each transformation, the loss of rank is listed. As the automatically selection takes the transformation for each time series with the best model accuracy rank, the loss of rank reﬂects the difference between the expectation of having the best forecasting rank and the actual forecast rank. For example, Log achieved the best rank for ANN and sARIMA over 44 time series based on model accuracy. When using the logarithmic transformation, however, ANN only achieves the best forecast for 10% of these time series. For all transformations and both methods, more than the half of the selected transformations achieves a rank worse than 1. Hence, the selection of the transformation based on the model accuracy achieves a rank worse than 2.5. These results underline Challenge 3. To this end, further and more intelligent approaches are required for achieving a better forecasting method recommendation.

Table 1.4: Rank loss after taking the best model transformation.

ANN sARIMA Loss Id Log Sqrt Box Normal Log Sqrt Box (2) (44) (10) (27) (0) (44) (11) (28) 0 0% 39% 10% 48% 0% 48% 9% 39% -1 0% 11% 20% 7% 0% 9% 27% 7% -2 0% 18% 50% 19% 0% 14% 64% 14% -3 100% 32% 20% 26% 0% 30% 0% 39%

1.6 Related Work

This section introduces related work of the ﬁelds (i) time series forecasting, (ii) feature engineering, and (iii) and auto-scaling. To the best of my knowledge, there is no forecast benchmark or a generic comparison method between forecasting methods in the literature.

1.6.1 Time Series Forecasting

In order to minimise the variance of single forecasting mechanisms and to face the ”No-Free-Lunch Theorem”, many hybrid mechanisms have been developed. These approaches can be grouped into three categories. The ﬁrst and oldest category is the ensemble forecasting. This technique was introduced by Bates and Granger in 1969 [2] and combines the results of at least two forecasting methods. Further works in this ﬁeld are form R. Clemen [8] and D. Menezes [11]. The second category uses forecast recommendation, where the best forecast method is guessed based on certain time series characteristics. The approaches either use an expert system [10] or machine 1 Challenges and Approaches: Forecasting for Autonomic Computing 17 learning techniques [28]. The last category uses the decomposition of time series to leverage additional information of a given time series. These approaches split the time series into linear and non-linear parts [31, 24, 20] or decompose the time series in its components (trend, seasonality, and noise) and/or further components [25].

1.6.2 Feature Engineering

In the scope of this work, two works are of interest. Khurana et al. [18] introduce a framework for feature selection and engineering. To this end, a transformation graph and reinforcement learning are used to explore suitable features and transformations. Further, J. Heaton [13] observed that the machine learning approach inﬂuences the features to consider as different methods can learn different kind of relationships.

1.6.3 Auto-Scaling

Lorido-Botran et al. [21] survey existing auto-scalers and propose a classiﬁcation of auto-scalers into ﬁve groups: (i) threshold-based rules, (ii) queueing theory, (iii) control theory, (iv) reinforcement learning, and (v) time series analysis. Further, Nikravesh et al. propose a kind of expert system to improve the prediction accuracy of auto-scaling [23].

1.7 Conclusion

Due to the fast-paced and changing requirements, software systems have a high degree of complexity. Thus, managing of some of these systems may exceed the human capacity. One solution is the usage of autonomic systems. In order to enable autonomic computing with proactive actions, reliable and accurate forecasting methods are required. Although forecasting is an important and established part of decision making, in the context of autonomic computing, there are challenges, such as a high variance in accuracy and time-to-result, that have to be faced. Thus, this work iden- tiﬁes 7 challenges (see Section 1.3) that have to be faced and proposes approaches to tackle them. The goals are, on the one hand, to develop a hybrid forecaster, which has robust performance, and, on the other hand, to establish a forecast benchmark for comparing different forecasting methods in a fair manner. Further, auto-scaling as use case for proactive autonomic computing is introduced. Based on the challenges, preliminary experiments are conducted. Besides the challenges, proposed solutions, possible extensions, and the preliminary experiments, a lot of work have to be done. Firstly, the evaluation of the proposed hybrid forecasting method have to extended as the preliminary experiments covers two 18 A. Bauer time series. Further, the method has to be enhanced to handle time series without seasonality. As autonomous systems have multiple sensors, an extension for multivariate time series is required. Although the results of the automatic feature transformation is currently not promising, further approaches have to be investigated.

References

1. Adhikari, R., Agrawal, R.K.: An introductory study on time series modeling and forecasting. CoRR abs/1302.6613 (2013) 2. Bates, J.M., Granger, C.W.: The combination of forecasts. Journal of the Operational Research Society 20(4), 451–468 (1969) 3. Bauer, A., Grohmann, J., Herbst, N., Kounev, S.: On the Value of Service Demand Esti- mation for Auto-Scaling. In: 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018). Springer (February 2018) 4. Bauer, A., Herbst, N., Spinner, S., Ali-Eldin, A., Kounev, S.: Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field. IEEE Transactions on Par- allel and Distributed Systems (September 2018), preprint 5. Box, G.E., Cox, D.R.: An analysis of transformations. Journal of the Royal Statistical Society. Series B (Methodological) pp. 211–252 (1964) 6. Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: ACM SIGKDD 2016. pp. 785–794. ACM (2016) 7. Chieu, T., more: Dynamic Scaling of Web Applications in a Virtualized Cloud Compu- ting Environment. In: IEEE ICEBE 2009. pp. 281–286. IEEE (2009) 8. Clemen, R.T.: Combining forecasts: A review and annotated bibliography. International Journal of Forecasting 5(4), 559–583 (1989) 9. Cleveland, R.B., Cleveland, W.S., McRae, J.E., Terpenning, I.: Stl: A seasonal-trend decomposition procedure based on loess. Journal of Ofﬁcial Statistics 6(1), 3–73 (1990) 10. Collopy, F., Armstrong, J.S.: Rule-based forecasting: Development and validation of an expert systems approach to combining time series extrapolations. Management Science 38(10), 1394–1414 (1992) 11. De Menezes, L.M., Bunn, D.W., Taylor, J.W.: Review of guidelines for the use of combined forecasts. European Journal of Operational Research 120(1), 190–204 (2000) 12. Guerrero, V.M.: Time-series analysis supported by power transformations. Journal of Forecasting 12(1), 37–48 (1993) 13. Heaton, J.: An empirical analysis of feature engineering for predictive modeling. In: SoutheastCon, 2016. pp. 1–6. IEEE (2016) 14. Herbst, N., Bauer, A., Kounev, S., Oikonomou, G., van Eyk, E., Kousiouris, G., Evan- gelinou, A., Krebs, R., Brecht, T., Abad, C.L., Iosup, A.: Quantifying Cloud Perfor- mance and Dependability: Taxonomy, Metric Design, and Emerging Challenges. ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToM- PECS) 3(4), 19:1–19:36 (August 2018), http://doi.acm.org/10.1145/3236332 15. Hyndman, R., Athanasopoulos, G., Bergmeir, C., Caceres, G., Chhay, L., O’Hara-Wild, M., Petropoulos, F., Razbash, S., Wang, E., Yasmeen, F.: forecast: Forecasting functions for time series and linear models (2018), http://pkg.robjhyndman.com/forecast, r package version 8.4 1 Challenges and Approaches: Forecasting for Autonomic Computing 19

16. Hyndman, R.J., Athanasopoulos, G.: Forecasting: principles and practice. OTexts, Mel- bourne, Australia (2014), OTexts.org/fpp 17. Kephart, J.O., Chess, D.M.: The vision of autonomic computing. Computer 36(1), 41–50 (2003) 18. Khurana, U., Samulowitz, H., Turaga, D.: Feature engineering for predictive modeling using reinforcement learning. arXiv preprint arXiv:1709.07150 (2017) 19. Kounev, S., Kephart, J.O., Milenkoski, A., Zhu, X. (eds.): Self-Aware Computing Sys- tems. Springer Verlag, Berlin Heidelberg, Germany (2017), http://www.springer.com/ de/book/9783319474724 20. Liu, N., Tang, Q., Zhang, J., more: A hybrid forecasting model with parameter optimization for short-term load forecasting of micro-grids. Applied Energy 129, 336–345 (2014) 21. Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A Review of Auto-scaling Tech- niques for Elastic Applications in Cloud Environments. Journal of Grid Computing 12(4), 559–592 (2014) 22. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 23. Nikravesh, A.Y., Ajila, S.A., Lung, C.H.: An autonomic prediction suite for cloud resource provisioning. Journal of Cloud Computing 6(1), 3 (2017) 24. Pai, P.F., Lin, C.S.: A hybrid arima and support vector machines model in stock price forecasting. Omega 33(6), 497–505 (2005) 25. Taylor, S.J., Letham, B.: Forecasting at scale. The American Statistician 72(1), 37–45 (2018) 26. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 27. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 28. Wang, X., Smith-Miles, K., Hyndman, R.: Rule induction for forecasting method selection: Meta-learning the characteristics of univariate time series. Neurocomputing 72(10 - 12), 2581 – 2594 (2009) 29. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans- actions on Evolutionary Computation 1(1), 67–82 (Apr 1997) 30. Wolpert, D.H.: The lack of a priori distinctions between learning algorithms. Neural computation 8(7), 1341–1390 (1996) 31. Zhang, G.P.: Time series forecasting using a hybrid arima and neural network model. Neurocomputing 50, 159–175 (2003) 32. Zuﬂe,¨ M., Bauer, A., Herbst, N., Curtef, V., Kounev, S.: Telescope: A Hybrid Fore- cast Method for Univariate Time Series. In: Proceedings of the International work- conference on Time Series (September 2017)

Predictive Maintenance in Complex Systems

Markus Gorlich¨

Organic Computing Group, University of Augsburg, Augsburg, Germany [email protected] Home Page: https://www.informatik.uni-augsburg.de/en/chairs/oc/

Abstract. Predictive Maintenance techniques are applied in a broad variety of technical systems in order to predict deterioration or failures of corresponding components. Concerning the increasing complexity in contemporary manufacturing paradigms, such as Industry 4.0, a brief overview on recent developments in the ﬁeld of Predictive Maintenance is given. Hereby, the focus lies on maintenance policy planning for multi-component systems as well as Ma- chine Learning-based approaches. Furthermore, possible starting points for future developments that could cope with this increasing complexity using Organic Computing techniques are discussed.

Keywords: Predictive Maintenance, Industry 4.0, Cyber-physical Systems, Industrial Inter- net of Things, Machine Learning, Complex Systems, Organic Computing

2.1 Introduction

Ever since manufacturing plants - or technical systems in general - were built, maintenance tasks have to be applied to them in order to keep them functioning according to their speciﬁcations. The overall industrial maintenance costs in the United States in the early 2000s exceeded $300 Billion per year [4]. Moreover, additional costs arise through machine downtimes caused by component failures or during the performance of maintenance tasks themselves. Consequently, it is of interest to minimise such costs. Predictive Maintenance (PdM) - also determined as Condition- based Maintenance - describes approaches aiming to lower these maintenance costs. In contrast to other maintenance policies, such as Preventive Maintenance, where components are replaced after some static time interval, PdM-techniques attempt to somehow predict the necessity of replacing or repairing a component. Such a prediction can be made, for example, by calculating a component’s detoration based on statistical information. 22 M. Gorlich¨

The recent advent of more complex manufacturing paradigms such as Industrial In- ternet of Things or Cyber-Physical Systems, summarised under the term Industry 4.0 [9], necessitate a more intelligent form of maintenance for several reasons. First of all, such manufacturing plants may consist of heterogenous production infrastructure from different suppliers. Also, new machinery may be integrated into the existing infrastructure from time to time. Finally, various dependencies between single machines or components may exist [19]. As existing PdM-techniques mostly focus on predicting maintenance information for single components, the development of novel maintenance paradigms, taking previously mentioned constraints into account, appears inevitable [21]. Secondly, contemporary manufacturing solutions may feature a higher number of sensors and other data sources that can be used to identify and predict component failures [23, 2]. Recent research activities attempting to use these vast amounts of data focus on Machine Learning-based solutions. Due to the dynamics of the maintenance problem, unanticipated events, and the presence of disturbances, the scenario is an interesting application domain for techniques from the Organic Computing domain [17, 28]. Several aspects have to be addressed to develop a fully scalable and efficient solution—ranging from basic system design [27] to interconnections [22] and to machine learning techniques [26, 24]. The goal of this paper is to provide a brief overview on research activities in the field of PdM for such complex systems [9]. Hereby, existing PdM-paradigms designed to cope with multi-component systems, regarded as global level, as well as Machine Learning-based approaches for single components, which can be viewed as local level are considered. Furthermore, research gaps and starting points for possible future work regarding such systems are identified. The remainder of this paper is structured as follows: Section 2.2 gives a brief overview on surveys and reviews related to the field of PdM. Existing multi-component approaches are examined in Section 2.2.1. ML techniques used for PdM are investigated in Section 2.2.2. Section 2.3 concludes with an outlook possible starting points for a PdM system for complex systems.

2.2 Related Work

Jardine et al. provide a review of various Condition-based Maintenance techniques in [8]. The paper focuses on techniques for data processing as well as diagnostic- and prognostic-based approaches for maintenance decision-making. Furthermore, different techniques for multiple sensor data fusion are discussed. Muller at el. provide various definitions regarding the concept of E-maintenance in [15]. A review of prognostic models used in Condition-based Maintenance, classified into the categories physical, knowledge-based, data-driven and combination models can be found in [20]. An overview of Time-based as well as Condition-based Maintenance techniques with focus on evaluating their actual applicability in industrial applications 2 Predictive Maintenance in Complex Systems 23 can be found in [1]. In [23], a contemporary evaluation with respect to changed conditions such as the availability of higher developed sensors or wireless networks is given. Besides the aforementioned reviews, which tend to provide a more general overview on existing techniques, various papers seeking to investigate the applicability of PdM techniques for specific application domains exist. In the following, current developments in PdM are presented. At first, approaches using PdM information to calculate cost-optimal maintenance policies for multi- component systems are discussed. Afterwards, current developments using Machine Learning techniques for PdM are introduced.

2.2.1 Maintenance Policy planning for Multi-Component Systems

An early approach towards PdM in multi-component systems is presented by Krause et al. in [11]. The authors claim that existing PdM-approaches cannot be applied in large-scale, heterogenous factory environments due to their composition of different machine types as well as machines from different manufactures. They propose a generic software framework focusing on three challenges: A seamingless integration of failure predictions into existing ERP-systems, a support for previously mentioned heterogenous factory floors as well as some sort of flexibility concerning the addition of new or removal of existing machines. However, the planning of actual maintenance policies is up to the ERP-system to which their PdM framework reports. An evaluation is not provided. An approach for maintenance policy planning with consideration of economic, structural and stochastic dependencies between components can be found in [29]. The authors attempt to minimise downtimes (and therefore maintenance costs) by grouping maintenance activities based on a stochastic degradation model for predicting component failures and previously mentioned dependencies between these components. The maintenance planning itself is done continuously based on the degradation predictions made. Compared with five conventional maintenance policies, the proposed approach shows significant cost savings due to its capability to dynamically react to changing component detoration. Nguyen et al. present an approach that focuses on the influence of a multi-component system’s structure on a single components deterioration in [18]. The authors make use of importance measures to identify important components and their influence on the whole system in order plan maintenance actions.

2.2.2 Machine Learning-based Predictive Maintenance

A popular Machine Learning technique used for PdM-tasks are Artiﬁcal Neural Net- works (ANN). Fu et al. use an ANN to predict the condition of a servo mechanism in a hydroelectric generating unit in [5]. A detailed evaluation of the proposed method is 24 M. Gorlich¨ not provided. An ANN as part of a system for minimising maintenance costs in rotational equipment is used estimate the life percentile and failure times of roller bearings in [30]. Their evaluation shows that the ANN’s predictions are mostly within ±10% of the actual life percentile. An approach using Convolutional Neural Net- works to abstract temporal dependencies between high-level states (therefore system states that can be observed by humans) from actual raw sensor data can be found in [7]. Although not speciﬁcally designed for PdM purposes, the authors claim that their approach could be applied to such problems as well due to the similarity of the used data. According to their evaluation, the proposed method performs superior to other suitable ANN-based approaches. A recent approach for vibration signal prediction in CNC machines using Deep Learning is proposed by Luo et al. in [14]. An average accuracy of 97.3% could be achieved. An comparison with other existing approaches is not provided. Other considerable approaches, for example using Principal Component Analysis [25], Bayesian Networks [6] or Genetic Algorithms [3, 13], that are not further discussed here due to the scope of this work, exist. Furthermore, approaches with focus on the analysis of gathered data using unsupervised techniques, such as Clustering [10] or Anomaly Detection [12], can be mentioned.

2.3 Future Work

In the following, potential starting points for future work concerning PdM in complex systems, taking the previously investigated approaches into account, are presented. As mentioned before, state-of-the-art manufacturing plants may feature a bright variety of sensors and other novel data sources as well as a heterogenous composition of machines from different types and manufacturers [23, 21]. The multi-component maintenance policies investigated in the previous section can be regarded as first steps towards mastering the complexity of such systems, taking dependencies between single components into account. However, they suffer from various drawbacks. First of all, the planning of such a maintenance policy itself is done by a single component, for example an ERP-system as in [11]. This is making the whole maintenance process dependent on the management component. A breakdown of the component (or, for example, a failure of the network infrastructure) might lead to a suspending of all maintenance activities until the failure is eliminated. Also, the presented maintenance policies do not take the vast amount of data sources that might be available into account: components (and the prediction of failures of those) are mostly modelled using stochastic degradation models. Collecting all available data sources in a centralised manner might lead to a challenging increase of complexity. Additionally, the use of PdM techniques might not be strictly limited to manufacturing plants (or Industry 4.0 systems) in the future. Mechanical components with sensors collecting data about their degradation can be found in a bright variety of devices, e.g. printers, cars or digital fitness devices. Although the (basic) mechanical 2 Predictive Maintenance in Complex Systems 25 technology remains the same, the kinds of harmful influences on these components may vary. Finally, due to the heterogenous nature of such systems, the degree of deterioration of a component may vary at runtime, for example due to frequent changes of a production cell’s task. In conclusion, three requirements for a PdM technique for complex systems can be formulated: (1) A decentralised way of constructing maintenance policies, allowing parts of a construction plant (or a complex technical system in general) to plan and request maintenance tasks independent from other components of the system. (2) A suitable approach to collect and aggregate data on a local level in order to be used for global maintenance planning. (3) Generic approaches to (local) Machine Learning as well as (global) maintenance planning in order to be applied to different kinds of technical systems as well as systems with changing conditions. A way to cope with these requirements lies in the field of Organic Computing (OC), in which increasing complexity in software systems should be resolved by autonomous, self-organised components and the interaction between themselves [16]. Possible OC-based solutions for PdM in complex systems could incorporate a self- organised construction of maintenance policies, taking local knowledge (dependencies, failure predictions) of each component into account. Autonomous subsystems (or components) could feature Online Machine Learning-techniques in order to be able to adapt to changing conditions within the system. Finally, OC concepts seek to move design-time decisions to runtime, which would allow a PdM-system to configure itself depending on the actual technical system it is applied to.

References

1. Ahmad, R., Kamaruddin, S.: An overview of time-based and condition-based maintenance in industrial application. Computers & Industrial Engineering 63(1), 135–149 (2012) 2. Bauernhansl, T., ten Hompel, M., Vogel-Heuser, B.: Industrie 4.0 in Produktion, Au- tomatisierung und Logistik. Springer Fachmedien Wiesbaden, Wiesbaden (2014) 3. Boudhar, H., Dahane, M., Rezg, N.: Spare part returns in stochastic deteriorating manufacturing system under a condition-based maintenance policy: Simulation-based Genetic Algorithm approach. IFAC Proceedings Volumes 46(9), 1399–1404 (2013) 4. Dhillon, B.S.: Engineering maintenance: A modern approach. CRC Press, Boca Raton FL (2002) 5. Fu, C., Ye, L., Liu, Y., Yu, R., Iung, B., Cheng, Y., Zeng, Y.: Predictive Maintenance in Intelligent-Control-Maintenance-Management System for Hydroelectric Generating Unit. IEEE Transactions on Energy Conversion 19(1), 179–186 (2004) 6. Gilabert, E., Arnaiz, A.: Intelligent automation systems for predictive maintenance: A case study. Robotics and Computer-Integrated Manufacturing 22(5-6), 543–549 (2006) 7. Guo, Y., Wu, Z., Ji, Y.: A Hybrid Deep Representation Learning Model for Time Series Classiﬁcation and Prediction. In: 2017 3rd International Conference on Big Data Com- puting and Communications (BIGCOM). pp. 226–231. IEEE (10082017 - 11082017) 26 M. Gorlich¨

8. Jardine, A.K., Lin, D., Banjevic, D.: A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing 20(7), 1483–1510 (2006) 9. Jeschke, S., Brecher, C., Meisen, T., Ozdemir,¨ D., Eschert, T.: Industrial Internet of Things and Cyber Manufacturing Systems. In: Jeschke, S., Brecher, C., Song, H., Rawat, D.B. (eds.) Industrial Internet of Things, Springer series in wireless technology, vol. 54, pp. 3–19. Springer, Cham (2017) 10. Kanawaday, A., Sane, A.: Machine learning for predictive maintenance of industrial machines using IoT sensor data. In: 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS). pp. 87–90. IEEE (24112017 - 26112017) 11. Krause, J., Cech, S., Rosenthal, F., Gssling, A., Groba, C., Vasyutynskyy, V.: Factory- wide predictive maintenance in heterogeneous environments. In: 2010 IEEE Interna- tional Workshop on Factory Communication Systems Proceedings. pp. 153–156 (May 2010) 12. Kroll, B., Schaffranek, D., Schriegel, S., Niggemann, O.: System modeling based on machine learning for anomaly detection and predictive maintenance in industrial plants. In: IEEE [International Conference on] Emerging Technologies and Factory Automation (ETFA), 2014. pp. 1–7. IEEE, Piscataway, NJ (2014) 13. Ladj, A., Varnier, C., Tayeb, F.B.S.: IPro-GA: an integrated prognostic based GA for scheduling jobs and predictive maintenance in a single multifunctional machine. IFAC- PapersOnLine 49(12), 1821–1826 (2016) 14. Luo, B., Wang, H., Liu, H., Li, B., Peng, F.: Early Fault Detection of Machine Tools Based on Deep Learning and Dynamic Identiﬁcation. IEEE Transactions on Industrial Electronics p. 1 (2018) 15. Muller, A., Crespo Marquez, A., Iung, B.: On the concept of e-maintenance: Review and current research. Reliability Engineering & System Safety 93(8), 1165–1187 (2008) 16. Muller-Schloer,¨ C., Schmeck, H., Ungerer, T.: Organic Computing — A Paradigm Shift for Complex Systems. Springer Basel, Basel (2011) 17. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 18. Nguyen, K.A., Do, P., Grall, A.: Condition-based maintenance for multi-component systems using importance measure and predictive information. International Journal of Sys- tems Science: Operations & Logistics 1(4), 228–245 (2014) 19. Nicolai, R.P., Dekker, R.: Optimal Maintenance of Multi-component Systems: A Re- view, pp. 263–286. Springer London, London (2008), https://doi.org/10.1007/978-1- 84800-011-7 11 20. Peng, Y., Dong, M., Zuo, M.J.: Current status of machine prognostics in condition-based maintenance: a review. The International Journal of Advanced Manufacturing Technol- ogy 50(1-4), 297–313 (2010) 21. Proenza, J., Willig, A. (eds.): WFCS 2010: 2010 IEEE International Workshop on Fac- tory Communication Systems proceedings May 18st-21rd, 2010, Loria, Nancy, France. Institute of Electrical and Electronics Engineers, Piscataway NJ (2010) 22. Rudolph, S., Hihn, R., Tomforde, S., Hhner, J.: Towards discovering delayed mutual inﬂuences in organic computing systems. In: ARCS 2017; 30th GI/ITG International Conference on Architecture of Computing Systems. pp. 39–46 (2017) 23. Selcuk, S.: Predictive maintenance, its implementation and latest trends. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 231(9), 1670–1679 (2017) 2 Predictive Maintenance in Complex Systems 27

24. Stein, A., Rauh, D., Tomforde, S., Hahner,¨ J.: Interpolation in the eXtended Classiﬁer System: An Architectural Perspective. Elsevier Journal of Systems Architecture (2017), http://www.sciencedirect.com/science/article/pii/S138376211730053X 25. Susto, G.A., Wan, J., Pampuri, S., Zanon, M., Johnston, A.B., O’Hara, P.G., McLoone, S.: An adaptive machine learning decision system for ﬂexible predictive maintenance. In: IEEE International Conference on Automation Science and Engineering (CASE), 2014. pp. 806–811. IEEE, Piscataway, NJ (2014) 26. Tomforde, S., Brameshuber, A., Hahner,¨ J., Muller-Schloer,¨ C.: Restricted On-line Learning in Real-world Systems. In: Proc. of the IEEE Congress on Evolutionary Com- putation (CEC11), held 05 Jun - 08 Jun 2011 in New Orleans, USA. pp. 1628 – 1635. IEEE (2011) 27. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 28. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 29. van Horenbeek, A., Pintelon, L.: A dynamic predictive maintenance policy for complex multi-component systems. Reliability Engineering & System Safety 120, 39–50 (2013) 30. Wu, S.j., Gebraeel, N., Lawley, M.A., Yih, Y.: A Neural Network Integrated Deci- sion Support System for Condition-Based Optimal Predictive Maintenance Policy. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 37(2), 226–236 (2007)

Towards an Agent-based Big Data Analytics Microservice System

Sebastian Proll¨

Distributed Systems Group, University of Augsburg, Augsburg, Germany [email protected] Home Page: http://dsg.hs-augsburg.de

Abstract. Current developments in data trafﬁc and data growth show how important and relevant a proper data handling strategy is, especially in industry and economy, where data is used as a business value creator. However, many organisations still struggle to keep up the pace of fast changing methods and technologies as well as the integration of data science components in their current system landscapes. Based on research activities of several case studies in an industrial manufacturing domain, four open research questions about designing and implementing big data analytics systems are raised and build the fundamentals for further studies and evaluations. To tackle these problems a highly dynamic, performable, scalable, extensible, available and deployable big data analytics microservice architecture is introduced in this paper. Concepts and methodologies of organic computing are applied to create a self-adapting and autonomous system by integrating software agents into the architecture design. This approach opens a way to further investigations of self-learning systems, using machine learning techniques to compose, manage and optimise the described agent-based big data analytics microservice system.

Keywords: Organic Computing, Software Agents, Big Data Analytics, Mircoservices, Indus- try 4.0.

3.1 Introduction

Digitisation with its industrial, economic, as well as ecological and social impact factors changes our daily lives. Industry 4.01 as one of its major drivers describes the fourth industrial revolution by generating innovations in connecting the physical and virtual worlds. By connecting more and more devices with the Internet, the data trafﬁc of machine to machine communication is prognosticated to 2.200 petabytes per month and the data trafﬁc of mobile devices to 42.000 petabytes per month [1].

1 also known as: Industrial Internet of Things or Smart Manufacturing 30 S. Proll¨

This huge amount of data traffic indicates an even bigger volume of stored data. Cisco Systems states that the volume of data would double every two years, predicting a volume of around 21 zetabytes in 2020 [2]. Additionally, the growing demands of productivity, flexibility and quality, as well as internal and external networking pushes industry and economy frequently beyond their limits. Profits can only be achieved with efficient technologies in production environments and management of the whole value chain. With so called cyber physical production systems (CPPS) manufacturing processes can be connected with virtual business processes to influence and manipulate the running production on all levels. Driven from that digitisation of CPPS the vision of industry 4.0 is of a) highly dynamic and individual production b) optimal use of resources to the production batch of one and c) interconnection of the whole value chain for real time analysis and monitoring [3]. To this point a lot of obstacles must still be tackled, especially when it comes to big data analytics (BDA) of running production processes [4, 5]. Due to this demand the field of machine learning, as a sub domain of artificial intelligence, emerges from rapidly increasing hardware performance and experienced another peak the last few years. More and more technologies evolve in the need of data analytics and big organisations such as Google, Microsoft, Facebook, Ama- zon and Netflix lead the way. However, small and medium-sized organisations often struggle to keep up the pace of this global competition. Not only due to the need of external support to handle these tasks, but also because they do not have the same data dimensions to work with. To deduce important impact factors to develop synergistic analytical systems, algorithms, tools and concepts a first goal is to identify common problems, solutions and best practices, which fit into the environment of highly individual production processes. Hereby, the focus lies mostly on implementing BDA solutions to intelligently collect and analyse mass data to optimise production processes consecutively and to design new data-based business model innovations. The combination of existing process-know-how with innovative scientific methods and services towards an industry 4.0-implementation is essential to tap the full potential of my current research activities. Driven from requirements to increase transparency in production processes, four use cases were identified to apply the described BDA system: 1. live control of the production 2. live quality monitoring in running production processes 3. optimisation of production scheduling 4. continuous product quality management Common BDA systems are highly distributed and demand consecutive adaption, optimisation and organisation. Design paradigms like Organic Computing (OC) provide approaches to develop methodologies and technologies for systems to gain the ability to self-adapt, self-organise, self-configure, self-optimise, self-heal and to gain awareness of themselves and their surroundings [6, 7, 8]. A popular methodology in OC is to design systems with software agents, wherein each agent represents one specific software or hardware component. Grouped in a multi agent system, various 3 Agent-based Big Data Analytics Microservice System 31 agents can cooperate, learn and mutate their solving strategies, system behaviours and evaluation process of problems. In the following paper, systems implementing the OC design paradigms are described as self-adapting or autonomous systems - as one of its key features in OC. A lot of research and related work was already done in the area of self-adapting and autonomous systems in production processes. Especially the concepts and methods of software agents [3] were introduced in industrial systems decades ago. The complex and individual system landscapes still render the integration of software agents a difficult task and it takes much effort to holistic integration and acceptance in the classic industry. However, the area of research of building self-adapting and autonomous distributed systems with software agents is thoroughly studied by the research community. Nevertheless, there are areas which bear potential for further research. An issue in data analytics projects is that data scientists rather work on demand - independently and separated from the rest of the organisation - than working from the beginning of the project as a fully integrated member. They have to analyse endlessly in historic datasets, already implemented systems and highly individual domain contexts. As a result, data scientists need approximately 80 percent of their time for business and data understanding, to be able to choose from a vast variety of methodologies to gain knowledge and insight of the underlying domain. Driven from current research activities and first hand experience in BDA projects four fundamental research questions (RQ) were raised: RQ 1 Is it possible to automate the process of business and data understanding to reliably choose relevant features and algorithms? RQ 2 Is it possible to automate, standardise and capture data analytics ensembles? RQ 3 Is there a way to design a reference model for data analytics systems in a semantic structure? RQ 4 Is it possible to automate the design process of self-adapting and highly individual data analytics systems? To describe the basic idea and concepts behind this approach the rest of the paper is structured as follows: Section 3.2 describes current challenges with data preprocessing in data analytics projects; Section 3.3 outlines the architecture of a highly dynamic, performable, scalable, extensible, available and deployable microservice system in the context of BDA systems; in Section 3.4 the concepts and methodologies of software agent-based systems is coupled with the idea of microservice systems to introduce a first approach of a software agent-based big data analytics microservice reference architecture for industrial production systems; finally Sec- tion 3.5 concludes current and further research activities. 32 S. Proll¨ 3.2 Challenges in Data Preprocessing

Current research activities have shown a big gap in actual data acquisition. Often data is collected, but not placed in the right context. Here, a methodology to collect data in a structured and easier processable way could be an event-based solution like event sourcing. Systems using the event sourcing paradigm promise extensive improvement in development and analytical processes. Event sourcing systems build their entire design on an event- and data-based structure. With this infrastructure, every process step can be reconstructed, if possible reversed or even simulated. Es- pecially in data analytics projects with predictive components the simulation feature is very promising.

Fig. 3.1: Big Picture Industry 4.0

However, many production processes are still missing very important metadata describing their semantic structure, leading to time consuming business and data understanding steps [10]. Unique identifiers, event types, synchronous timestamps for accurate traceability, event sources and maintained formats, standards or specifications are just a few important information data scientists require to be able to work efficiently [4, 9]. Not only data scientists profit from a coherent data infrastructure. System analysts, engineers, developers and designers are not only able to understand 3 Agent-based Big Data Analytics Microservice System 33 data flows better, but can also build new system components upon that specified data structure. In this way a whole new business and system awareness can emerge, just by ”speaking the same language”. In general, the first step in data science projects is to gain fundamental understanding of underlying business processes, domain specific knowledge and their connection to persisted information in form of data. The vision of industry 4.0 describes a completely interconnected industry system containing various data sources like MES, ERP, PLM systems, etc. . Different types of data sources are categorised in historical data like data stored in different kinds of SQL and NoSQL databases, (live) event data e.g. system events, sensor data or results of data processing- and learning-algorithms (see Fig. 3.2). With so many different data sources, each type of data and different format (structured, semi-structured or unstructured), it takes a lot of effort to implement an operational solution. Especially when it comes to analytical approaches the effort spent in (semantic) data preparation escalates very quickly. Without standards in data formats and semantic structures, e.g. Portable Format for Analytics (PFA) or Predictive Model Markup Language (PMML)2, applicable data lake solutions, infrastructures and current industry applications are hard to implement. Current investigations show that many organisations in the industrial production sector already handle their production processes more or less event-based, but most of them have no proper event processing implemented yet. To develop a in general applicable event infrastructure and to lay the foundation of the agent-based BDA system the earlier described use cases are studied to find commonalities between specific event patterns, alarms and system failures.

3.3 Flexible System Architectures with Microservices

Most of current system landscapes are still too unstructured and heterogeneous. In modern BDA systems, it is necessary to develop a flexible, dynamic and scalable software system, which helps to fine tune current and future design and technology decisions [10, 11]. Following methodologies for data mining and data fusion is essential for developing a BDA platform, for instance IBM‘s Analytics Solutions Unified Method for Data Mining/Predictive Analytics (ASUM-DM) as an advancement of the well- established Cross-Industry Standard Process for Data Mining (CRISP-DM), as well as the Joint Directors of Laboratories (JDL) model [12]. Referring to the CRISP- DM and the JDL model the architecture as illustrated in Fig. 3.2 was designed and is currently under implementation. Similar to the CRISP-DM our architecture design is separated into four categories: Data Source, Data Vortex, Data Streams and Data Lake.

2 developed by the Data Mining Group: http://dmg.org (last access: 04/02/2019) 34 S. Proll¨

3.3.1 Data Source

The first category represents the data integration with various data sources. Here, the focus lies on establishing access to the data. This includes authentication/authori- sation, connection to databases, file system access, connection to data streams and access to cluster nodes in case of distributed server or file storage.

3.3.2 Data Vortex

In order to develop a data model for further analysis, evaluation and deployment the data has to be pre-processed, which is covered by the second category. In the so called data vortex, the data is integrated, discovered, validated, cleansed, transformed, and reﬁned for better understanding and to improve further proccessing. This step is crucial because of its importance to gain an understanding of correla- tions and relationships and to assign data to its correct domain context.

3.3.3 Data Streams

Suitable data is provided as a data stream, e.g. time series data or series of events. Before persisting the pre-processed data, that data can additionally be correlated, reﬁned and analysed, using complex event processing (CEP) techniques with tools like Apache Flink and/or data stream analytics with tools like Apache Spark. This process complements the BDA architecture design.

3.3.4 Data Lake

Is the data pre-processed and ready to persist, it can be stored into a so-called data lake. Since big data is handled, this data lake is most possibly based on frameworks like Apache Hadoop, allowing a clustered, high-performance and scalable environment to store data in a specified format, e.g. JSON format. In the data lake the data itself should follow a standardised format, like the earlier introduced PFA. The data lake is the basis that allows further processing for analysis tools, visualisation tools, as well as standard business tools, like MES, ERP and PLA systems. As our current research activities have shown, a proper visualisation and monitoring of information is crucial. Very often insights to specific production processes can easily be gained by merely looking at the data in question in different types of diagrams. At this point the challenge is, that in every link of the value chain different kinds of information are important and thus have to be placed in different context views. That is why a lot of attention should be paid developing modular dashboard applications mirroring on-demand information in a customisable way. Data models, fitting to possible data 3 Agent-based Big Data Analytics Microservice System 35 analytics algorithms, tools and frameworks, can be developed by using the underlying and (semantically) prepared data from the data lake. The results can than be further processed, played back into running production and business applications or visualised, for example in mobile applications, dashboards and monitoring systems. The key concept in this approach lies in the data and event communication between the four categories. With a system-wide standard representation of semantically enriched data and events, all different kinds of system components speak the same language. This makes it easier to develop, integrate and replace system components.

Fig. 3.2: Big Data Analytics Architecture

Developing such large systems with microservices that involve the whole value chain improves not only earlier described benefits [13]. Dividing the system into so called bounded contexts (domain-specific boundries) allows a strong modularisation with replaceable system components. This sets the foundation for a sustainable, agile and dynamically scalable development process. By linking one specific task to a microservice (bounded context), developers can work highly independent, because each microservice contains all relevant layers, including user interface, backend and data storage. Additionally, different technology stacks can be used for each microservice, which allows the combination of an ideal environment for each context. To interconnect microservices, these simply have to agree on standardised interfaces like OPC-UA, REST or message-based solutions like MQTT. If one microservice is not conform to the given requirements any more, it can easily be replaced by another mi- 36 S. Proll¨ croservice, that implements the same interface. Innovative solutions like containeri- sation, e.g. Docker containers, allow a fully automated continuous deployment and continuous delivery process of individual microservices, reducing the time to market tremendously [13]. Zimmermann et al. describe a basic blueprint of an enterprise services architecture reference model. Based on the Open Group Architecture Framework this blueprint provides solutions for an adaptive model, capable of supporting manual integration decisions with automated digital systems [14]. Vakali et al. [16] also introduce a platform for real time data analytics (REAλICS) implementing the λ-architecture and Chen et al. [20] deliver a detailed architecture centred methodology for BDA systems, evaluated in ten case studies. These solutions are focused on a decision support system for digital architecture management of non-analytics system components focusing on performance, scalability, extensibility, availability and deployability [20]. Goal of my future research will be to transfer these concepts on data analytics system components. First approaches for intelligent BDA were already done by Siriweera et al. [10] using nested automatic service composition (NASC) to automate the CRISP- DM model. However, they still manually processed the first two steps of business and data understanding, which still has to be further studied (RQ 1). A major problem is to structure business information and data semantically to be able to automate these steps. Further research activities will study and evaluate this approach. Elshambakey et al. [17], Kantere et al. [18] and Villanustre et al. [19] show different approaches by designing and orchestrating data analytics applications in a workflow model allowing them to react and adapt more flexible. In these approaches, the integration of domain specific (meta)information and structuring these semantically, are not focused as target interests, opening a niche I would like to explore in my doctoral thesis. This could be accomplished via technologies like ontologies, allowing to structure information in a machine interpretable way. Different approaches are already using ontologies for self-adapting industrial production systems, but they do not work with analytical system components, leaving space for RQ 2 and RQ 3.

3.4 Self-Adapting Systems

Self-adapting systems, as in the vision of industry 4.0, are often described as intelligent systems. ”Intelligent” means, that the very products, machines and decisions relating to the processes can be manipulated in real or near-real time. These tasks are mostly accomplished self-adapted utilising technologies like software agents and CEP. An additional characteristic of intelligent systems is the usage of ”intelligent products”. The idea is to be able to identify and locate products in the production at all times. Hereby, they also provide (semantically prepared) information [21, 22] about them, their current state and if there are possible alternative ways to reach their final state [15, 23]. Small RFID tags, for instance, can be attached to products and 3 Agent-based Big Data Analytics Microservice System 37 resources in order to make them intelligent (semantic product memory) [15]. Even- tually, machines and products gain intelligence by the use of sensors, the ability to be programmed, to communicate and collaborate with each other and act accordingly. With software agents, as a machine’s virtual representation, new ways of intelligent and autonomous production systems are can be implemented [24, 25]. Grouped in multi agent systems they can plan, schedule and execute processes without direct interaction with or manipulation by users [22]. Employing software agents emerged as a way to build large autonomous systems that react to information and occurring events, for example e-commerce [21] or industrial production systems [24, 3]. Here, a software agent can not only represent, for example, specific machines, but also business processes and system users, often called actors in this context [3, 22]. A common methodology for multi agent systems is to publish tasks on a so called blackboard. This blackboard is shared amongst various agents and responsible agents can commit themselves to tasks, preferably in a semantical representation, and try to collaborate in a solving strategy [21]. Obitko et al. show how distributed agent-based solutions in industrial systems can be enriched with semantics and ontologies [27]. As already mentioned, implementations of semantic-enriched and self-adapting applications are already to be found, especially in industrial production environments [3, 22, 24, 26]. Agents using CEP technologies can gather and analyse important information in large systems to gain insights about their surroundings. CEP engine rules can be defined to filter data streams and aggregate events accordingly to the engine rules. Complex events can then be used to trigger actions or receive information about the system status. In BDA systems, CEP methodologies can be used to analyse streaming data of time series- or event-based data. Apache Spark for instance, has established a high performance [29, 33] and broadly used [32] big data framework to execute mass data operations. On top of that, it offers several tools to apply machine learning algorithms on the data streams [28, 30, 31], which can be exploited to complement the agents decision making process. With machine learning tools and frameworks to learn and predict different kinds of production situations, software agents are able to make their decisions more accurate, more dynamic and faster. Examples would be forecasts of production stoppages and breakdowns or live optimisation of production scheduling, to name just a few. Whereas, in classic multi agent systems the agents’ decisions are mostly based on specific rules. Working with machine learning approaches enhances the agents’ intelligence. However, this approach still needs further research and is also addressed by RQ 4. To provide answers to this question, further study of the key concepts, methodologies and technologies to model a semantic data analytics agent system is planned. The in Fig. 3.3 described multi agent system is being developed in another research project at the university of applied sciences Augsburg and illustrates a smart distributed software agent system for an industrial production environment. Enhanced 38 S. Proll¨

Fig. 3.3: Data Analytics Agent System with components of a BDA system, this is a first design approach to RQ 4. Along with the existing agents to run the distributed production system, e.g. contact, order, workpiece and machine agent, the agents relevant to data processing and analysis have yet to be implemented. These agents could consist of a short term memory (STM) analytics agent for live data and event analysis, a long term memory (LTM) analytics agent for analysis of historic data and events, a CEP agent for complex event processing, a data agent for further data processing tasks like cleansing or aggregating data and an OWL agent to provide semantic information. The semantic information is about production tasks, analytics tasks, relationships and structures of system components, as well as of data and events. To predict and react in near real time to system faults and delays the STM analytics agent is responsible for on-premise analysis like stream data analytics. Possible scenarios are: failure classification with pre-trained neural networks, reinforcement learning neural networks or live system-halt predictions with a Markov model for instance. On the other side, the LTM Analytics Agent takes advantage of historical data. It is responsible for trend analysis and long term forecasts, e.g. production schedule or business process related insights like resource planing for just-in-time deliveries or customer behaviours like sales forecasts. These are just a few examples for the vast fields of application for data analytics agents. In collaboration these two short and long term types of analytics agents are capable of building dynamic and powerful data analytics ensembles, 3 Agent-based Big Data Analytics Microservice System 39 combining various strategies and algorithms to gain insights of the underlying system. In the described multi agent system directly collaborating agents are grouped together in one microservice. This concept prevents major performance issues, like network traffic and data sharing. With the STM analytics agent on-premise analysis reduces data traffic, by exchanging only relevant and preprocessed data and events. Due to the start-up phase of my research activities, agent-based BDA microservice systems and its concepts, technologies and suitable methodologies have to be further studied and evaluated. Therefor Vikas [26] presents an interesting concept of a multi agent cognitive architecture for mobile edge computing based on microservices, trying to answer similar questions as stated in RQ 4.

3.5 Conclusion

In this paper, current research activities towards an agent-based BDA microservice system were introduced. Major problems that are seen in the development of BDA projects for industrial production processes include: missing data handling, metadata and semantic structure of data; time consuming business and data understanding phases; the need of dynamic analytics ensembles; automated and self-adapting management of system services. The system in Fig. 3.2 was designed to develop a flexible, scalable and synergistic microservice architecture with data analytics components. Complemented with the potential of multi agent systems, the data analytics agent system is to be further investigated and integrated into the system design of the BDA microservice architecture. Amongst other things, latest developments in data science and machine learning promise new solutions in this context. Inspired by the four research questions introduced in Section 3.1 a semantic representation of business, system, infrastructure and communication aspects will be developed. Specifi- cally, business and production tasks for business understanding, data and event structures for data understanding and algorithm input/output for combinations of various data analytics algorithms are interesting topics for continuing research. OC techniques are taken into account for the process of autonomous feature detection and algorithm selection. Also an user-centred approach is planned, which covers a scientific study of the user requirements of the four introduced use cases. In collaboration with industrial manufacturing organisations the four in Section 3.1 introduced use cases are used to collect different requirements, concepts and ideas of an agent-based BDA microservice system and evaluated with real life data. With these collected, knowledge topologies of the earlier described aspects can be developed. Being able to use that information for autonomous systems, the topology has to be translated into an ontology. With technologies like Web Ontology Language (OWL3) the information can be interpreted by software (agents) and serves as a foundation to reason over specific questions in running production and business processes. Based on the

3 https://www.w3.org/TR/owl2-overview/ (last access: 04/02/2019) 40 S. Proll¨ results of the semantically enriched BDA system I strive for an approach to automate the business and data understanding phases of data analytics systems.

References

1. Cisco Systems: Cisco Visual Networking Index. Prognose zum Datenvolumen des Datacenter-Prognose zum monatlichen Datenverkehr von portablen Geraten¨ in Mobilfunknetzen weltweit von 2016 bis 2021 nach Geratetyp¨ (in Petabyte), https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking- index-vni/mobile-white-paper-c11-520862.pdf, (2017) 2. Cisco Systems: Cisco Global Cloud Index. Prognose zum Datenvolumen des Datacenter-Trafﬁcs weltweit in den Jahren 2016 bis 2021 (in Zetabyte), https://www.cisco.com/c/en/us/solutions/collateral/service-provider/global-cloud- index-gci/white-paper-c11-738085.pdf, (2018) 3. Vogel-Heuser, B. (ed.): Softwareagenten in der Industrie 4.0. De Gruyter Oldenbourg, Berlin, Boston (2018) 4. Libes, D., Shin, S., Woo, J.: Considerations and recommendations for data availability for data analytics for manufacturing. In: Ho, H. (ed.) 2015 IEEE International Confer- ence on Big Data. Oct 29-Nov 01, 2015, Santa Clara, CA, USA : proceedings, pp. 68-75. IEEE, Piscataway, NJ (2015) 5. Nino, M., Saenz, F., Blanco, J.M., Illarramendi, A.: Requirements for a big data capturing and integration architecture in a distributed manufacturing scenario. In: 2016 IEEE 14th International Conference on Industrial Informatics (INDIN). Proceedings : Palais des Congrs du Futuroscope, Futuroscope - Poitiers, France, 19-21 July, 2016, pp. 1326- 1329. IEEE, Piscataway, NJ (2016) 6. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 7. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January2017), http://arxiv.org/abs/1701.08125 8. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 9. Lade, P., Ghosh, R., Srinivasan, S.: Manufacturing Analytics and Industrial Internet of Things. IEEE Intell. Syst. 32, 74-79 (2017) 10. T. H. A. S. Siriweera, I. Paik, B. T. G. S. Kumara, K. R. C. Koswatta (eds.): Intelligent Big Data Analysis Architecture Based on Automatic Service Composition. 2015 IEEE International Congress on Big Data (2015) 11. Braun, E., Schlachter, T., Dupmeier,¨ C., Stucky, K.-U., Suess, W.: A Generic Microser- vice Architecture for Environmental Data Management. In: Hrebˇ ´ıcek,ˇ J., Denzer, R., Schimak, G., Pitner, T. (eds.) Environmental Software Systems. Computer Science for Environmental Protection. 12th IFIP WG 5.11 International Symposium, ISESS 2017, Zadar, Croatia, May 10-12, 2017, Proceedings, 507, pp. 383-394. Springer International Publishing, Cham (2017) 12. Waltz, E.: Fundamentals of Reasoning and Multisensing. In: Multisensor Fusion (ISBN: 978-94-010-0556-2), Springer (2002) 3 Agent-based Big Data Analytics Microservice System 41

13. Wolff, E.: Microservices. Grundlagen flexibler Softwarearchitekturen. dpunkt.verlag, Heidelberg (2016) 14. Zimmermann, A., Schmidt, R., Sandkuhl, K., Jugel, D., Bogner, J., Mohring,¨ M.: Decision-Controlled Digitization Architecture for Internet of Things and Microservices. In: Czarnowski, I., Howlett, R.J., Jain, L.C. (eds.) Intelligent Decision Technologies 2017. Proceedings of the 9th KES International Conference on Intelligent Decision Technologies (KES-IDT 2017) - Part II, 73, pp. 82-92. Springer International Publishing, Cham, s.l. (2018) 15. Ferrarini, L., Veber, C., Luder,¨ A., Peschke, J., Katogeras, A., Gialelis, J., Rode, J., Wunsch,¨ D., Chapurlat, V.: Control architecture for reconfigurable manufacturing systems. The PABADIS‘PROMISE approach. In: ETFA 2006 proceedings, pp. 545-552. IEEE Operations Center, Piscataway, NJ (2006) 16. Vakali, A., Korosoglou, P., Daoglou, P.: A multi-layer software architecture framework for adaptive real-time analytics. In: Joshi, J., Data, I.I.C.o.B. (eds.) 2016 IEEE Interna- tional Conference on Big Data. Dec 05-Dec 08, 2015, Washington D.C., USA : proceedings, pp. 2425-2430. IEEE, [Piscataway, NJ] (2016) 17. M. Elshambakey, M. Khalefa, W. J. Tolone, S. Das Bhattacharjee, H. Lee, L. Cinquini, S. Schlueter, I. Cho, W. Dou, D. J. Crichton (eds.): Towards a distributed infrastructure for data-driven discoveries and analysis. 2017 IEEE International Conference on Big Data (Big Data) (2017) 18. V. Kantere, M. Filatov (eds.): A Workflow Model for Adaptive Analytics on Big Data. 2015 IEEE International Congress on Big Data (2015) 19. F. Villanustre (ed.): Industrial Big Data Analytics: Lessons from the Trenches. 2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering (2015) 20. Chen, H.-M., Kazman, R., Haziyev, S.: Agile Big Data Analytics for Web-Based Sys- tems: An Architecture-Centric Approach. IEEE Trans. Big Data 2, 234-248 (2016) 5. Dayyani, B.: Software architecture design and development of multi-layer highly modular platform using intelligent components for dynamic big data analytics. In: ISCBI (ed.) 2016 4th International Symposium on Computational and Business Intelligence. ISCBI 2016 : September 05-07, 2016, Olten, Switzerland, pp. 45-53. IEEE, Piscataway, NJ (2016) 21. Wooldridge, M.J.: An introduction to multiagent systems. Wiley, Chichester (2009) 22. Kacprzyk, J., Eli, A., Kon, M.T., Orgun, M.A.: Semantic Agent Systems. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) 23. Seitz, C., Scholer¨ Thorsten, Neidig Jorg¨ (eds.): An Agent-based Sensor Middleware for generating and interpreting Digital Product Memories. AAMAS 8th International Conference on Autonomous Agents and Multiagent Systems (2009) 24. Gohner,¨ P.: Agentensysteme in der Automatisierungstechnik. Springer Berlin Heidel- berg, Berlin, Heidelberg (2013) 25. Ishida, T., Jennings, N.R., Sycara, K., Bussmann, S., Wooldridge, M.: Multiagent Sys- tems for Manufacturing Control. Springer Berlin Heidelberg, Berlin, Heidelberg (2004) 26. Shah, V.S.: Multi-agent cognitive architecture-enabled IoT applications of mobile edge computing. Ann. Telecommun. 73, 487-497 (2018) 27. K. Nagorny, S. Scholze, M. Ruhl, A. W. Colombo (eds.): Semantical support for a CPS data marketplace to prepare Big Data analytics in smart manufacturing environments. 2018 IEEE Industrial Cyber-Physical Systems (ICPS) (2018) 28. A. Londhe, P. P. Rao (eds.): Platforms for big data analytics: Trend towards hybrid era. 2017 International Conference on Energy, Communication, Data Analytics and Soft Computing (ICECDS) (2017) 42 S. Proll¨

29. A. Unnikrishnan, U. Narayanan, S. Joseph (eds.): Performance analysis of various supervised algorithms on big data. 2017 International Conference on Energy, Communication, Data Analytics and Soft Computing (ICECDS) (2017) 30. Veiga, J., Exposito, R.R., Pardo, X.C., Taboada, G.L., Touriﬁo, J.: Performance evaluation of big data frameworks for large-scale data analytics. In: Joshi, J., Data, I.I.C.o.B. (eds.) 2016 IEEE International Conference on Big Data. Dec 05-Dec 08, 2015, Wash- ington D.C., USA : proceedings, pp. 424-431. IEEE, [Piscataway, NJ] (2016) 31. N. W. Grady, J. A. Payne, H. Parker (eds.): Agile big data analytics: AnalyticsOps for data science. 2017 IEEE International Conference on Big Data (Big Data) (2017) 32. Demchenko, Y., Laat, C. de, Membrey, P.: Deﬁning architecture components of the Big Data Ecosystem. In: Smari, W.W., Fox, G.C., Nygrd, M. (eds.) Proceedings of the 2014 International Conference on Collaboration Technologies and Systems. May 19-23, 2014 : the Commons Hotel, Minneapolis, Minnesota, pp. 104-112. IEEE, Piscataway, NJ (2014) 33. K. Park, M. C. Nguyen, H. Won (eds.): Web-based collaborative big data analytics on big data as a service platform. 2015 17th International Conference on Advanced Com- munication Technology (ICACT) (2015) 34. M. V. Nural, H. Peng, J. A. Miller (eds.): Using meta-learning for model type selection in predictive big data analytics. 2017 IEEE International Conference on Big Data (Big Data) (2017) Part II

Machine Learning as Basis for Organic Computing

Smart Device Based Initial Movement Detection of Cyclists Using Convolutional Neural Networks

Jan Schneegans and Maarten Bieshaar

Intelligent Embedded Systems Lab, University of Kassel, Kassel, Germany {jschneegans,mbieshaar}@uni-kassel.de

Abstract. For future traffic scenarios, we envision interconnected traffic participants, who exchange information about their current state, e.g., their position or predicted intentions, allowing them to act in a cooperative manner. Vulnerable road users, e.g., pedestrians and cyclists, are equipped with smart devices that can be used to detect their intentions and transmit these detected intentions to approaching cars such that the drivers can be warned. In this article, the focus lies on detecting the initial starting movement of cyclists using smart devices. Smart devices provide the necessary sensors, namely accelerometer and gyroscope, and therefore pose an excellent instrument to detect movement transitions (e.g., waiting to moving) fast. Convolutional Neural Networks prove to be the state-of-the-art solution for many problems with an ever increasing range of applications. Therefore, we model the initial movement detection as a classification problem. In terms of Organic Computing (OC), this presents a step towards self-awareness and self-adaptation. We apply residual network architectures to the task of detecting the initial starting movement of cyclists.

Keywords: Machine Learning, Human Activity Recognition, Context Awareness, Self-Adaptation, Self-Awareness, Organic Computing, Intention Detection, Deep Learning, Convolutional Neural Network, Residual Network

4.1 Introduction

We envision future mixed traffic scenarios [1], involving automated cars, trucks, sensor-equipped infrastructure, and other road users equipped with smart devices and wearables. As these wearables are interconnected by means of an ad hoc network, they can cooperate on different levels, e.g., on the level of detected intentions. Vulnerable road users (VRUs) will continue to take part in future urban traffic, hence recognising them and detecting their intentions poses an important goal towards a safer traffic environment. Many vehicles already incorporate a range of sensors, e.g., 46 J. Schneegans and M. Bieshaar

Radar or Lidar, but they rely on an unhindered line of sight. The adoption of smart devices implements an alternative considering most people already carry one with them while participating in traffic. Such devices come pre-equipped with the necessary hardware such as accelerometer, gyroscope and magnetic field sensors and in near future will be capable to connect with their surroundings via ad hoc networks. While GPS sensors enable only a rough localisation and do not offer the necessary sampling rates to detect fast movement changes, systems based on inertial data allow for a fast detection of changes in motion. In this article, we focus on the reliable detection of starting movement transitions, i.e., initial movement detection. On the one hand, the detector has to be robust to avoid false positive detections, and on the other hand, it has to be fast. As was shown in [2], the initial movement detection can greatly support the VRU’s trajectory forecast. In terms of Organic Computing [10, 15, 14], the system requires self-reflection capabilities in order to adapt subsequent algorithms, e.g., for trajectory forecasting. In this article, an approach to smart device based initial movement detection is presented aiming to bring self-reflection capabilities and context awareness to smart devices in the domain of VRU intention detection. The approach presented in this article aims to detect starting movements of cyclists fast and yet robust. The approach is based on a convolutional neural network trained on the inertial data (accelerometer and gyroscope) obtained from a smart device worn by the cyclist. In Section 4.2, the related work is reviewed. Section 4.3 presents our approach including the preprocessing pipeline and the applied deep learning methods, as well as the evaluation methodology. Following a description of the data acquisition, Sec- tion 4.4 demonstrates the experimental results. Section 4.5 concludes the results and provides further insight into future work.

4.2 Related Work

Previous work focused on the detection of human activities for various applications ranging from elder care, to the detection of motion types like standing, walking up or down stairs etc., or even footstep detection [13]. Thereby, the use of machine learning techniques plays an increasingly dominant role. Employing Deep Convolutional Neural Networks (CNNs), rather than handcrafted features, appears to pose a viable option based on the success of this emerging technology. Applying machine learning to perform human activity recognition based on smart phone motion and sensor data is presented for example by Jiang and Yin [9]. Their data is given by the accelerometer and gyroscope of wearable devices. Deep Convolutional Neural Networks enable them to automatically learn the optimal features for the activity recognition task, providing state of the art results on data from wearable sensors. The key feature of their success relies on so-called activity images, which are created out of multiple 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 47 tracks of sensor data that are permuted so that a neighbouring of every pair of tracks is given, followed by a Fourier Transform. Using the activity images as input, their machine learning model is then able to learn features connecting information contained in multiple tracks of data. The learned features in a convolutional network thereby substitute handcrafted features which otherwise have to be extracted independently from multiple time series sensor signals. This indicates the possibility to detect further motion primitives, such as starting and stopping motions, in a traffic environment utilising smart device data. Bieshaar et al. [3] envision a cooperative traffic environment where each individual is connected to the surrounding traffic participants and infrastructure, while maintaining a model of the present situation. They propose a system where video data from cameras at intersections and smartphone data are combined via a boosted stacking ensemble. This combination leads to a robust classification, although being a complex procedure. Their approach was able to detect 99% of the starting movements within 0.24 ms after the first movement of the bicycle’s back wheel. The system is comprised of a 3D-CNN for the image processing and a smart device-based starting movement detector consisting of an extreme gradient boosting classifier (XGBoost). Working on a similar topic, Zernetsch et al. [18] operate solely on camera data fed as Motion History Images into a Residual Network (ResNet) and a Support Vector Ma- chine (SVM). Both approaches are capable of a correct classification, whereby the ResNet is more robust and accurate than the SVM. In another article [17], they built a forecasting model predicting the location of vulnerable road users for the next 2.5 s. The experiments include two methods: a physics-based computational model of a cyclist and a polynomial least-squares approximation in combination with a multi- layer perceptron. Both were compared to a Kalman filter approach and improved on its results. The physics-based model achieved a 27% more accurate positional prediction, while the least-squares approximation attained a 34% increase in accuracy for the starting and stopping motion of a cyclist with otherwise similar outcome as the first approach. Their results underline the applicability of deep learning models to the movement detection task. Further work regarding the smart device wearing location is done by Bieshaar et al. [4]. In recent work, multiple wearing locations including the trouser pocket, jacket and backpack (mounted on the bicycle or worn by the cyclist) were tested. Having fewer false positive detection, putting the smart device in the trouser pocket is deemed the most reliable approach. Also combining classifiers for different wearing locations improves the results compared to having one model handling all devices simultaneously.

4.3 Methodology

Our approach aims at detecting the movement transition between waiting and moving (i.e., starting) of cyclists as early as possible using a smart device. The starting 48 J. Schneegans and M. Bieshaar movement detector is realised by means of an HAR pipeline [5]. A schematic of the starting movement detector consisting of three stages is depicted in Fig. 4.1. The cyclists starting movement is modeled as a classiﬁcation problem of the classes waiting, starting and moving. Inertial data, i.e., accelerometer and gyroscope, is used as input for the starting movement detector. The data is preprocessed (i.e., transformed into a device attitude invariant representation) followed by a feature extraction. Fi- nally, the detection is realised by means of a deep learning based classiﬁer taking the form of a residual neural network.

Fig. 4.1: Processing pipeline consisting of three stages: (1) preprocessing, (2) feature extraction and feature selection, and (3) classiﬁcation using convolutional neural network taking the form of a residual neural network.

4.3.1 Preprocessing and Feature Extraction

The first modules of the starting movement detector consist of the data preprocessing and feature extraction. We compare four different sets of features. The raw data is provided by the accelerometer and gyroscope of a smart device. Be- fore feeding it into the model, the x, y and z axis are rotated to lie flat on the earth, i.e., the z-axis pointing towards the sky. This is done with the help of rotational information provided by a magnetometer inside the smart device. To avoid the tedious estimation of device-to-VRU transformation, we consider the magnitude of the x and y axis. This leaves four features to be used: the gravity-adjusted magnitude in the x-y-plane and the z-component of the accelerometer and gyroscope measurements. This comprises the first set of features. A schematic of this procedure is depicted in Fig. 4.2. On the preprocessed data a Fourier Transform up to the fifth degree was constructed based on a sliding window segmentation. It provides coefficients and phases over windows of size 640 ms and 5120 ms. This comprises the second set of features. The preprocessed data was further handled to generate over 150 features, including the mean, variance, energy, min, max and polynomial characteristics over window sizes of 100 ms, 500 ms, 1000 ms and 2000 ms. Those features (including the Fourier coefficients) were sieved through a filter approach consisting of four procedures as depicted in Fig. 4.3. To obtain a large diversity concerning the selected features, four complementary types of filters were picked. Two feature selection techniques: minimum redundancy maximum relevance (MrMR), mutual information (MIFS); and 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 49

Fig. 4.2: Two-step pre-processing of the raw sensor signal. First attitude information is used to rotate x, y and z value of the accelerometer and gyroscope, then the magnitude of x and y is calculated. two model based methods applying an elastic net (ElasticNet) and extreme gradient boosted trees (XGBoost). A ten fold cross-validation was executed and the ten most indicative features of each feature selector were extracted for each fold separately. A union set of those selected features was then utilised, amounting to roughly 19 features per fold. This third set of features will be referred to as ﬁltered features.

Fig. 4.3: The filter approach pipeline reducing the selected features. Four classifiers select the 10 best features and thereafter a union produces the final features set.

Moreover, the Fourier coefﬁcients and phases were evaluated for each sensor (accelerometer, gyroscope) and window (5120 ms, 640 ms) separately. Also a combination of the ﬁlter features and the Fourier phase was tested to see if further improvements could be achieved. They were therefore simply concatenated creating the largest of input spaces containing 31 features and comprising the fourth set of features.

4.3.2 Classiﬁcation using Residual Networks

The last stage of the starting movement detector was realised by means of deep learning, i.e., a residual neural network, which is a special kind of convolutional neural network. Deep learning has been recently proven to be extremely successful in various domains and has lead to an increased usage of machine learning techniques in a multitude of ﬁelds. Convolutional neural networks (CNN) have already been applied to many practical tasks and have risen in popularity after achieving superhuman accuracy on image classiﬁcation tasks [12]. They work by sequentially applying convolutions over an input space and are trained via back-propagation of error gradients. Their input space 50 J. Schneegans and M. Bieshaar typically takes the form of a 2D or 3D image with depth greater or equal to one. Stacking multiple layers of convolution helps to identify broader features and covers more of the input space the deeper the network grows, enabling the model to learn abstract characteristics regarding the data.

4.3.2.1 Residual Networks

In standard convolutional networks a layer is only connected to the preceding and proceeding layers. This limits the information visible to deeper layers to activations of the preceding layer, i.e., only features captured by the previous layer are exploited by the current layer. An architecture getting around this constraint is the Residual Network (ResNet) proposed by He et al. [6]. A network comprised of this ResNet architecture won the 1st place of the ImageNet Large Scale Visual Recognition Compe- tition [12] in 2015. It was originally designed to conquer the problem of degradation in very deep networks. When deeper networks start converging, accuracy reaches a saturation point, before degrading rapidly with increasing network depth. To resolve this issue, the authors introduced shortcut connections skipping convolutional layers and therefore providing deeper layers with the identity of the original input. Instead of learning a direct mapping of H(x) : x → y, a residual function F(x)=H(x) − x is deﬁned. It can be reformed into H(x)=F(x)+x, where F(x) and x represent the non-linear layers, e.g., convolution, activation or regularisation layers, and the identity function respectively. According to the ﬁndings of He et al., the residual function F(x) is easier to optimise than H(x). For example when trying to learn the identity function, the shortcut connection x can be facilitated and F(x) can easily be learned to be pushed to zeros by a stack of non-linear convolutional layers. Residual Unit: The residual unit is a computational unit comprising the identity shortcut x and the residual function F(x), which itself is composed of convolutional, activation and batch normalisation layers [8]. He et al. strongly suggest to apply them in the order of batch normalisation, activation function followed by the convolution [7]. The best position of the shortcut is around two sets of layers ordered as just mentioned . A schematic of a residual unit is depicted in Fig 4.4.

4.3.2.2 Regularisation

Regularisation methods aim at decreasing the validation and test error, thus increasing the generalisation abilities of machine learning models. In the following the regularisation schemes adopted in this work will be described.

Spatial Dropout

Dropout is used in neural networks to hinder the network from being too reliant on the same subset of learned features. A dropout probability (sometimes given in the 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 51

Fig. 4.4: Schematic of a residual unit composed of the identity shortcut x (left) and the residual function F(x) (right) transforming the input from the previous layer xl to the input of the next layer xl+1. form of the complementary keep probability) is introduced that randomly zeros a proportion of neurons in the network. Therefore, the usage of features is distributed more evenly and the network learns to use more robust combinations of features. Spatial dropout is a special form of dropout being used in convolutional networks, motivating the learning of more independent features. In contrast to normal dropout, where individual weights are excluded, whole ﬁlters are set to zero. Since a ﬁlter is comprised of multiple weights, this mimics the original intent of omitting complete and randomly selected features during training more closely than just discarding parts of it. Although spatial dropout increases the temporal requirements while training, it encourages the learning of more robust features, consequently leading to a smaller validation and test error, i.e., better generalisation ability.

Batch Normalisation

To speed up the learning process and increase the generalisation ability, batch normalisation can be applied. It is implemented by adding a special layer before a convolution which normalises the activation of the previous layer according to the batch statistics, i.e., the mean and variance of the current mini-batch [8]. This works because it minimises the amount of covariance shift of the hidden units. As an example, consider a network trained on black cat images: When applying this network on colored cats, it will perform worse, because while it still depicts a cat, the distribution of color values is different. In other words, if an algorithm learned some X to Y mapping, and if the distribution of X changes, then we might need to retrain the learning algorithm by trying to align the distribution of X with the distribution of Y. Batch normalisation is favoring a more stable learning of features by normalising the in- 52 J. Schneegans and M. Bieshaar puts to deep hidden layers reducing the effect of covariance shift, because the mean and variance remain similar during training [11]. Additional beneﬁts of batch normalisation include being able to use higher learning rates, since the input values to the convolutional layers are restricted from being very high or very low. It also adds a regularisation effect by adding noise to the activations (due to the statistics being calculated only on the current mini-batch).

4.3.2.3 Ensemble Properties

Since the early days of machine learning ensemble techniques are used to improve performance. In [16], Veit et al. unravel residual networks and conclude that they exhibit ensemble-like properties, largely due to being composed of multiple paths of various length throughout the network. To underline this statement, they deleted layers from different conventional convolutional network architectures, which reduced their performance to chance levels. Only the ResNet proved to be resilient to such alteration as its error margin increased only slightly.

4.3.3 Architecture

For the third and last module of the starting intention detector, the classifier, two architectures of the ResNet were examined during the experiments. They are depicted in Fig 4.5. The smaller 8-layer network (Fig. 4.5, top left) consists of two arrays of residual units preceded by a first convolutional layer. The first array holds two residual units with eight filters, while the second array holds one residual array with 16 filters. For the second array, the dimensions are halved. An additional convolution is used to adjust the dimensions of the data for the shortcut between the first and second array. The final fully connected layer reduces the spatial dimensions to the desired output size of three classes, followed by a softmax layer, which is applied to obtain values in the interval [0,1]. The larger 14-layer network (Fig. 4.5, right) follows the same architectural approach but with three arrays holding two residual units each. This results in 14 layers, also counting the first convolutional and last fully connected layers. The first layer and array both begin with 16 filters, doubling the number of filters and reducing the spatial dimensions by one half every array, just as in the smaller model. The models were trained by optimising the weighted cross-entropy applying the Adam-Algorithm. As indicated earlier, spatial dropout and batch normalisation were applied. The residual unit was structured as explained in Section 4.3.2.1 with the addition of a dropout layer right before each convolution (excluding the first convolutional layer), as can be seen in Fig. 4.5 (bottom-left). The convolutions were carried out with 3x3 filters, stride set to one and zero padding. 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 53

4.3.4 Evaluation

We model the starting movement detection problem as a classification problem. It consists of three classes indicating waiting, starting and moving, which split a time series in three parts. This separation is depicted in Fig. 4.6. The most common label is waiting (I) and describes an interval of low movement and acceleration, while the cyclist is waiting, e.g., at a traffic light. An initial movement that leads to a start, including the starting motion up to the moving phase, is labeled as starting (II). Con- sequently, moving (III) is defined as the first movement of the back wheel followed by an acceleration and driving.

Fig. 4.5: Architecture of the networks used: (top-left) small network consisting of eight layers and starting with eight filters. (right) larger network consisting of 14 layers starting with 16 filters. Each shortcut connection (black curved arrows) encompasses two arrays (grouping) of layers, each comprising batch normalisation, activation function (Rectified Linear Unit), spatial dropout and convolution as depicted in the bottom-left.

In order to calculate scores on the validation and test data the F1-score was used. It is calculated from the precision and recall, which in turn are calculated from the true- positive, true-negative, false-positive and false-negative values defined as follows: if the first occurrence of a moving prediction is after the true starting label, it is classified as a true-positive. This means that a classification of a start is still classified as a true positive even if it is detected before the moving label but during the initial 54 J. Schneegans and M. Bieshaar

Fig. 4.6: In red: example probabilities for the moving class of a starting prediction. Area I encompasses the waiting class of low activity; II denotes the starting class meaning the initial motion leading to the start, beginning at time s; III marks the moving class labeled after the first movement of the bicycles back wheel, beginning at time m. movement of the test person intending to start. Else if it is too early, i.e., before the first true starting label, it is classified as a false-positive. True-negatives normally do not appear since every time series contains starting motion. When the predictor would not output the starting class at any time step, it is classified as a false-negative. The delay of the starting prediction is determined only for true positive classifications, as otherwise false positives would greatly reduce the average measured delay. It is calculated relative to the first true moving label. Negative values are possible, because a classification during the starting phase is still considered a true-positive.

4.4 Experiments

The experiments were separated into two parts: First, a feature selection phase, where each of the four feature sets were evaluated with the smaller 8-layer network; and second, an architecture optimisation phase, where both the smaller 8-layer and the larger 14-layer models were evaluated on the feature set selected in the previous feature selection phase. On each data set a ten fold cross-validation was performed and the average scores on each fold were combined to create ﬁnal results over the whole data set. In order to compensate for the unbalanced distribution of the three classes (e.g. the waiting class occurs much more often than the starting class) a class weighting was applied in the loss function of the network. The weights are 0.2, 0.7 and 0.1 for the moving, starting and waiting classes, respectively. Throughout the feature selection experiments, input windows of sizes 0.2 s, 1 s and 2 s were tested with a ﬁxed keep probability of 0.7 for the spatial dropout layers. 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 55

After the feature selection experiments, the ﬁlter features and an input window of 1 s were selected to be used in the architecture optimisation, during which multiple values (0.6, 0.7, 0.8) for the keep probability of the spatial dropout layers were tested with both network sizes.

4.4.1 Data Acquisition

The data of 49 test persons was gathered during a week long experiment at a public intersection, incorporating bicycle lanes as well as walkways. It consists of 84 start scenes and the ground truth was generated through a wide angle stereo camera system and visual inspection [3]. The cyclist were instructed to cross the road or drive alongside it, adhering to the trafﬁc rules. Each carried a smart phone (Samsung Galaxy S6) in the right trouser pocked. Earlier work indicates the trouser pocket as the most promising location to wear the smart device [4].

4.4.2 Feature Selection

First, the feature selection was conducted. The results are displayed in Fig. 4.7. To explain the names, take for example the filter phase lin acc 5120 100 model: it uses the filtered feature plus the Fourier phase (on an input window of 5120 ms) of the linear accelerometer sensor (linear means gravity adjusted) and takes the 100 previous time steps (2 s, with 50 Hz) as input. The results turned out as expected: a smaller input window leads to lower delays, a larger input window leads to higher delays. The convolutions themselves add to the delay as they cover multiple time steps on higher level learned features. The filtered features lead to higher delays compared to the raw features, as the additional features such as mean or variance have to be calculated over windows as well. Nonetheless, they are more robust and provide a more stable score over different thresholds applied on the final softmax output.

The highest F1-scores are achieved by the filtered features. On an input window of 2 s it reaches an F1-score of 95.1% and has a mean delay of 861 ms. With an input window of 1 s it has a slightly faster mean delay of 783 ms. The raw features end up in the middle field, yet, not as good as the filtered features with an input window of 0.2 s. With an input window of 2 s the highest F1-score is 76.3%, having a very low mean delay of 257 ms. Combining the filter features with the phase of a discrete Fourier Transformation reduces the scores compared to only filter features. We suppose that the 8-layer network is too small to deal with so many features. Using the discrete Fourier Transform coefficients and phases on their own does not provide sufficient information for a starting intention detection. The highest F1-score of 62.7% was reached by the Fourier coefficients (over a window of 5120 ms) of the Accelerometer with an input window of 2 s. All other Fourier features are worse, especially the phase reaching only a maximum F1-score of 32%. 56 J. Schneegans and M. Bieshaar

Fig. 4.7: Results of the feature selection sorted by delay (left) and F1 score (right). Through the feature selection, a keep probability of 0.7 was applied. The combination of feature set and input window size that was selected for further examination in the architecture optimisation is marked in bold. It uses the ﬁlter features on an input window of 1 s, having a mean F1-score of 96.6 % and a mean delay of 783 ms. 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 57

4.4.3 Architecture Optimisation

For the architecture optimisation, the filtered features and an input window of 1 s were selected as a compromise between the classification score and delay. Also an input window of 0.2 s is quite prone to false positives, since there is often movement while waiting, e.g., switching the leg on which the cyclist rests, preparing the ped- als or simply looking around. For the same reasons, filtered features are preferred over the raw data, because spikes of activity are smoothed out over the feature windows. An overview over the results of the architecture optimisation are displayed in Fig. 4.9.

Fig. 4.8: Classification results (left) and Kernel Density Estimation of the delay (right) of the selected model, which processes the filter features, has an input window of 50 time steps and keep probability of 0.6. Most starting classifications occur between 0.5 s and 1 s after the moving label. Noticeably, some starting detections happen almost 1 s before the bicycle moves.

The bigger model did not reach the same performance on the test set as the smaller one, having a maximum F1-score of 92.6% with a keep probability of 0.8. This is sup- posedly because of over-fitting since the training loss was comparable to the smaller models. Nonetheless, the mean delay times improved with the model size, reaching a minimal mean delay of 592 ms with a keep probability of 0.7. A decisive conclusion on the keep probability could not be drawn due to no apparent trend for it better being high or low. An overview of the performance of a selected model can be seen in Fig. 4.8. This model achieved the highest F1-score of all trained models. It processes the filtered features, has an input window of 1 s and keep probability of 0.6. On the left, the precision, F1-score and mean detection time (delay) are plotted against multiple threshold values, that were applied on the final softmax layer of the network to the moving class. At a threshold value of 0.5 it reaches an F1-score of 96.6% and mean delay of 783 ms. On the right, a kernel density estimation over all delays is displayed; this includes every true-positive starting detections. The center of this distribution lies 58 J. Schneegans and M. Bieshaar around the mean detection time of 783 ms. It is noticeable though, that some detections happen almost 1 s before the bicycle moves.

Fig. 4.9: Results of the architecture optimisation sorted by delay and F1 score. Through the feature selection, the ﬁlter features and an input window of 1 s were chosen. A model with a keep probability of 0.6 provided the best overall performance.

Through Fig. 4.10, where the F1-score is plotted against the mean detection time for all trained models, the Pareto optimality can be examined. The dots are colored according to the different data sets that were evaluated. The ﬁltered features that were chosen for the architecture optimisation (orange) produce the highest F1-score. Noticeably, the raw features (blue) are also Pareto-optimal, due to the low delay. A positive correlation between the input window size and delay, as well as the F1-score, can be seen. With increasing window size the F1-score grows, but so does the delay. It is desired to ﬁnd a model on the Pareto-frontier, outlined by the Pareto-optimal solutions, that compromises both target variables.

4.5 Conclusion and Future Work

The high scores on the selected models show that the use of convolutional neural networks on sensory data from smart phones is justified. Our approach reaches a maximum F1-score of 96.6% with an average delay of 783 ms. Considering the Pareto optimality, the raw features separate themselves with a low delay, but are not as reliable. Deep Residual Neural Networks prove to be applicable to the task of starting intention detection. They achieve a reliable classification, although with a compara- tively high delay of over half a second. Further improvements can be achieved with more training data. Cutting the data to not include stopping motion greatly improved scores, since very high excitations are then only encountered during the starting motion. This could be implemented with a separate waiting detector, which activates the starting detector only when the stopping motion was finished and switching back to it once driving. The next step is to utilise the results of this work and produce a trajectory forecast for cyclists, based fully on smart devices. Furthermore, utilising multiple smart devices carried in different location, possibly integrated into the helmet, shoes or mounted on the bicycle, will likely improve reliability as well as shorten reaction times. Our 4 Smart Device Based Initial Movement Detection of Cyclists Using CNNs 59

Fig. 4.10: Mean delay and F1 plotted against each other for the different feature sets. approach will be generalised to different VRUs, such as pedestrians, enabling the integration of multiple sensor sources into one cooperative environment. To reduce the dependency on camera systems for labeling, active and semi-supervised learning techniques can be applied to incorporate imprecise labels. This will also simplify our investigations in continuous online improvements of the learned models.

Acknowledgment

This work results from the project DeCoInt2, supported by the German Research Foundation (DFG) within the priority programme SPP 1835: ”Kooperativ inter- agierende Automobile”, grant number SI 674/11-1.

References

1. Bieshaar, M., Reitberger, G., Zernetsch, S., Sick, B., Fuchs, E., Doll, K.: Detecting Inten- tions of Vulnerable Road Users Based on Collective Intelligence. In: AAET. pp. 67–87. Braunschweig, Germany (2017) 2. Bieshaar, M., Zernetsch, S., Depping, M., Sick, B., Doll, K.: Cooperative Starting Inten- tion Detection of Cyclists Based on Smart Devices and Infrastructure. In: ITSC. Yoko- hama, Japan (2017) 60 J. Schneegans and M. Bieshaar

3. Bieshaar, M., Zernetsch, S., Hubert, A., Sick, B., Doll, K.: Cooperative starting movement detection of cyclists using convolutional neural networks and a boosted stacking ensemble. Trans. on IV 3(4), 534–544 (2018) 4. Bieshaar, M., Depping, M., Schneegans, J., Sick, B.: Starting Movement Detection of Cyclists Using Smart Devices. In: DSAA. Turin, Italy (2018) 5. Bulling, A., Blanke, U., Schiele, B.: A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors. ACM Comput. Surv. pp. 1–33 (2014) 6. He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: CVPR. Las Vegas, NV, USA (2015) 7. He, K., Zhang, X., Ren, S., Sun, J.: Identity Mappings in Deep Residual Networks. In: ECCV. Amsterdam, The Netherlands (2016) 8. Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: ICML. pp. 448–456. Lille, France (2015) 9. Jiang, W., Yin, Z.: Human Activity Recognition using Wearable Sensors by Deep Con- volutional Neural Networks. In: Proc. of International conference on Multimedia (MM). pp. 1307–1310. New York, NY, USA (2015) 10. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 11. Ng, A.: Why does Batch Norm Work? (C2W3L06). Deeplearning.ai https://www.youtube.com/watch?v=nUUqwaxLnWs (2017), (Last accessed: 28.05.2018) 12. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. IJCV (3), 211–252 (2015) 13. Singh, M.S., Pondenkandath, V., Zhou, B., Lukowicz, P., Liwicki, M.: Transforming Sensor Data to the Image Domain for Deep Learning - an Application to Footstep De- tection. In: IJCNN. Anchorage, AK, USA (2017) 14. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 15. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 16. Veit, A., Wilber, M., Belongie, S.: Residual Networks Behave Like Ensembles of Rela- tively Shallow Networks. In: NIPS. pp. 550–558. Barcelona, Spain (2016) 17. Zernetsch, S., Kohnen, S., Goldhammer, M., Doll, K., Sick, B.: Trajectory Prediction of Cyclists Using a Physical Model and an Artiﬁcial Neural Network. In: IV. Gothenburg, Sweden (2016) 18. Zernetsch, S., Kress, V., Sick, B., Doll, K.: Early Start Intention Detection of Cyclists Using Motion History Images and a Deep Residual Network. In: IV. Chang Shu, China (2018) 5

Transfer Learning for Optimisation of Carbon Fiber Reinforced Polymer Production

Simon Stieber

Institute for Software & Systems Engineering, University of Augsburg, Augsburg, Germany [email protected] Home Page: https://www.isse.uni-augsburg.de/

Abstract. The main problem that keeps many areas of research from using Deep Learning methods is the lack of sufﬁcient amounts of data. We propose transfer learning from simulated data as a solution to that issue. In this work, we present the industrial use case for which we plan to apply our transfer learning approach to: the production of economic Carbon Fiber Re- inforced Polymer components. It is currently common practice to draw samples of produced components statistically and perform destructive tests on them, which is very costly. Our goal is to predict the quality of components during the production process. This has the advantage of enabling not only online monitoring but also adaptively optimising the manufacturing procedure. The data comes from sensors embedded in a tooling in a Resin Transfer Molding press.

Keywords: Transfer Learning, Simulation-based Machine Learning, Organic Computing, Carbon Fiber Reinforced Polymer

5.1 Motivation

Machine Learning (ML) and especially Deep Learning (DL) algorithms have been successfully applied to certain fields of research recently: Computer Vision [11] or Speech Recognition [2]. Moreover, Reinforcement Learning (RL) in combination with Deep Learning (DL) was used to solve games (e.g., Go [22] and Atari games [15]). Other areas, for instance, industrial production or medical sciences, try to benefit from these algorithm families as well. The problem that keeps those and many other areas from adapting DL to the respective problems is the immense need for data when using deep neural networks. The gathering of enough data and the subsequent labeling phase is costly or can be almost impossible for certain kinds of data. The use case we present is the automation, optimisation, and adaptation of industrial production of Carbon Fiber Reinforced Polymer (CFRP) that should be improved by 62 S. Stieber transfer learning from simulated data. Components made of Carbon fiber make vehicles lighter by 50-70 % compared to conventional steel and therefore more energy- efficient: A 1 % weight reduction reduces the energy consumption by approximately 0.6-0.7 % [6]. Currently, the production of CFRP is too expensive for automotive volume scale, because it is by far more complicated to produce CFRP than press steel or aluminum [23]. The factors that drive the cost the most are difficult automation, high cycle times, personnel, industrial manufacturing equipment, and material cost. In low-scale production industries such as avionics and aerospace, even more cost-inefficient factors such as increased manual labour and intermediate products, called “carbon pre-pregs”1 increase costs even further. Our solution addresses several of these problems: we use an automated Resin Transfer Molding (RTM) process (see Fig. 5.1) with more cost-efficient materials: non-wovens made of recycled carbon fibers and a small share of natural fibers and caprolactam as resin.

Sensors:

Temperature

Electric A B Acoustic

Pressure

Actuators: C Press B Time 3. 2. D E 1. F

4. 4.1

Fig. 5.1: Schematic Resin Transfer Molding (RTM) press with symbolic sensors and actuators.

This combination of an Resin Transfer Molding (RTM) process with non-wovens and online in-situ monitoring during the production process is unique to our knowledge

1 Pre-preg stands for “pre-impregnated” composite fibers. Within these, a polymer matrix material, e.g., epoxy, is already present in the fiber. 5 Transfer Learning for Optimisation of CFRP 63 and makes the whole endeavour very complex. The manufacturing of components for mere training purposes is non-economical, since they are disposed of thereafter. The press and its components are shown in Fig. 5.1. A and B are containers for the two-component resin. The mixing unit is marked with C. D illustrates the press. The carbon mat E is about to be injected with resin. F represents the tooling that holds the negative of the future component and is equipped with acoustic, electric, temperature and pressure sensors. The process steps are: 1. The non-woven carbon mat is brought into the press, which has around 100 degrees Celsius. 2. Resin is injected into the mold. 3. The Press is pressed and resin spreads in the mat. 4. Sensors measure the spread distribution and the hardening of the resin. 4.1. If necessary, the press keeps shut for a longer period or additional pressure is applied. 5. The press is opened and the finished component is extracted. In this position paper, we present the initial goals of the project and suggested paths towards them. We propose learning on simulated data (as in robotics [21] or gam- ing [22, 15]) and refining that knowledge with few real-world samples as a solution. While the dataset of real-world samples within our project is too small for DL by a magnitude, as many simulated runs as necessary can be produced. Additionally, the time-consuming data labeling step is no longer necessary since the data is already labeled when it comes from the simulation: all stages of resin distribution and curing are available for every spot of the component. Researching transfer learning from simulated to real data is a major focus of our work. We want to identify and tackle the Machine Learning (ML) challenges at the gap between simulation and the real world. The smart Carbon Fiber Reinforced Polymer (CFRP) press that is constructed during my dissertation is self-learning and self-adaptive. These self-X capabilities are a key concept of organic computing systems [4]. The restore invariant approach [17] from the field of Organic Computing [16, 24] that describes the corridor of correct behaviour can be applied to this machine. Initially, the press has to be parametrised to get into this corridor of correct behaviour: it produces CFRP components with a high quality. Whenever the quality is declining—moving out of the corridor—the process is adapted by the machine itself. In Section 5.2 we present the expected challenges and the goals of the project. Sec- tion 5.3 entails recent works on transfer learning on the one hand and works on non-destructive testing of CFRP components via acoustic and electric sensors on the other hand. Methods, algorithms, and ideas on how to tackle the challenges of this project are shown in Section 5.4. The last part, Section 5.5 gives an overview of future research activities. Additionally, our work is put into context with respect to our project partners’ contributions. 64 S. Stieber 5.2 Objectives

Bridging the gap between simulated flow fronts2 and real sensor data will be a major challenge. Grssing et al. [3] compared the simulation software we are going to use, PAM RTM3, a subset of PAM Composites by ESI, and an open source alternative with regards to how realistic they model flow fronts. During the work on the project, we will test how suitable PAM RTM is for our setup. One key question is how to merge the two types of data. Are we going to be able to learn on raw sensor data, time series of ultrasonic and electric sensors, or do we use abstracted flow fronts from real data for training? Both types of training data have to be aligned to make it possible to coherently train on them. The research on transfer learning from simulated and real data is an important aspect of my dissertation. Partners in the project have been working on similar questions in the past. Kalafat and Sause [7] sensed damages in carbon components with acoustic sensors. In our project, similar sensors, as well as piezo-electric sensors, will be used to sense the distribution of the resin in the carbon fiber mat and the polymerisation. Another challenge is to handle varying environmental factors such as air temperature and humidity. These can interfere with the production progress: especially the caprolactam is sensitive to humidity. In a controlled production environment, such environmental factors should become tamable. Carbon long fibers, the raw material, have a variance in weight per meter of 10 %4. That makes the material’s behaviour less predictable than that of steel or aluminum. Variations in the production material will be a central issue. In addition to the variance in weight of the raw material, a mix of recycled carbon and natural fibers will be used to produce the non-woven mats for our project. Different distributions of these materials in the non-wovens will cause different permeabilities. Permeability is a very important property when predicting the distribution of the resin in the mat. We want to tackle this challenge by varying the input parameters of the simulation to cover problematic non-woven mats. The goals for our part in this project, that build on top of each other, are: 1. Classifying the quality5 of one component from mere sensor input 2. Optimising the quality over the production process of multiple components

2 The flow front is the edge between the part of the mat that has been impregnated with resin and the rest that has to be impregnated yet. The flow front simulation software is able to simulate the distribution process of the resin. 3 https://https://www.esi-group.com/de/software-loesungen/virtual-manufacturing/ composites/pam-composites/pam-rtm (last access: 29/08/2018) 4 Source: Voith Composites, http://voith.com/composites-de/index.html (last access: 15/11/2018) 5 The quality of a CFRP component is determined by several factors. The resin has to be distributed equally and has to reach all parts of the fiber mat. Additionally, no air or other materials shall be enclosed in the component. Then the stability is optimal. 5 Transfer Learning for Optimisation of CFRP 65

3. Get a real-time prediction of one component as it is still in the press and adjust certain parameters if the outcome is expected to be sub-optimal, see Fig. 5.1, step 4.1. Furthermore, the optimisation of the algorithm is part of our work, as soon as the goals regarding functionality are met. The main priority is to ensure that the inference step meets the requirements for online adaption of parameters at production speed. The hard time limit needs to be determined first, once the timing of the process is clear. As soon as the timing of the process is defined, we can set the specifications for our algorithm and start optimising.

5.3 State of the art - Related work

There have been preliminary works on optimising the production of CFRP products. Sorg [23] compared several data mining algorithms on data coming from the production of a carbon car roof. The sources of the data were already established process logs which held data about several time spans, thicknesses of the product at certain points, temperatures, and pressures. Within our project these sorts of data will be available as well as data that is specifically gathered for our needs: sensor data that shows the flow front and curing progress. These acoustic and electric sensors have already been used to detect damages in CFRP components in-situ on one hand. Kostka et al. [10] and Larrosa et al. [12] both have been working on monitoring the health status of an already finished CFRP component with sensors (acoustic and electric), that were embedded in the component during production. Kostka et al. used decision trees and Larrosa et al. implemented Gaussian discriminative analysis to reach their goals. Kalafat and Sause have shown that the localisation of defects in CFRP via neural networks is feasible and outperforms other techniques [7] . Opposed to these authors that used sensor data to determine the health status of a CFRP component, we will use acoustic and electric sensors to determine the flow front and the curing process within in the press. Their works are important with regard to using sensors to determine the state of a component already in use. A research company named Synthesites, on the other hand, has published several works on online monitoring of the production of CFRP. Pantelelis et al. [19] have shown that it is possible to monitor not only the distribution of the resin but also the curing of the product with electrical and temperature sensors. In another work which was carried out in a EU project6, they have shown that they can use neural networks for the optimisation of the curing process [28]. In their work, the heating profile for a CFRP component was optimised so that production could be accelerated

6 IREMO (intelligent REactive polymer composites MOulding) https://cordis.europa.eu/ project/rcn/94018 en.html (last access: 22/08/2018) 66 S. Stieber by 36 % with 5 % less heating energy involved compared to no curing monitoring and control. They used very shallow neural networks with only one hidden layer and used bootstrap aggregating to combine the output of 30 networks. Besides the data from online monitoring and the neural networks involved, there is a third similarity of their work to our project: they used simulated data for initial training. We delimit our work from theirs by several points. First, the online monitoring in our project concerns not only the distribution of resin and curing of the component at several points of interest, but the exact flow front process, damage detection during deforming, and monitoring of temperature, pressure, and acoustics in the tooling. We operate on a much larger scale regarding data compared to the other project: a sensor network made of many more sensors will be used in our project. They used only five single sensors. We will use more sophisticated flow front simulations that are specially made for CFRP production refined with data coming from the aforementioned sensor network. Fr the other works, a simpler, one-dimensional simulation method by Pantelelis et al. [18] was used. Another difference lays within the utilised materials: we use caprolactam as resin and non-woven carbon mats whilst they were using epoxy and wovens. As mentioned in Section 5.2, Grossing¨ et al. compared PAM RTM, and OpenFOAM7 regarding their ability to model flow fronts. To test how realistic the simulations are, they installed a transparent upper half on an RTM press to make the flow front advancement visible. In contrast to our work, they used a resin with high viscosity and carbon pre-pregs. Their experiments showed that PAM RTM is computationally more efficient because of the different underlying mathematical model. It is easier to operate, but has the disadvantage that it cannot model the transition between porous and non-porous/solid material. Once we start using PAM RTM, we will see if this issue is relevant to our work and, if so, we need to check if that issue is solved in a recent version of the software since the paper appeared in 2016. After covering the related work concerning CFRP, the following part will present research on learning methods that have been utilised for other use-cases. We expect for at least some of these approaches to work well in the context of CFRP production. Weiss et al. [25] give a comprehensive and in-depth overview of the methods of transfer learning. They present different domains where algorithms of this sort have been implemented successfully, i.e. for image classification, human-activity classification, and software defect classification. They do not explicitly treat simulation-to- real data transfer learning but give a good overview of common techniques to adapt the knowledge of an algorithm from one domain to the other. The main method to do so is to fine-tune a neural network, a method that is described more thoroughly in Section 5.4. Progressive Neural Networks [20] are another interesting idea to handle the transfer of knowledge from one domain to another. They have been used for transferring knowledge from simulation to real data in a robot use case [21], which makes them well suited for our problem. Section 5.4 gives an introduction to the operating principle behind this type of network.

7 https://www.openfoam.com/ (last acces: 29/08/2018) 5 Transfer Learning for Optimisation of CFRP 67

Other sources focus on different applications of DL algorithms to specific use cases with the help of simulations. These include traffic flow prediction [14] and the understanding of dynamical scenes from video data [26]. Both works have shown that it is possible to learn from simulated data and to adapt that knowledge to real-world applications. They lack the link to CFRP or industrial production in general, a gap we plan to fill.

5.4 Methodology

Initially, we will train our model to predict the quality (our definition of quality is described in item 1) of a small and simple carbon component. In a second step, a chicane component that exhibits a complex geometry, for instance ascents or round edges, will be produced. These three steps are necessary due to the fact that the production equipment will be installed in parallel to the first steps in the project and the final system will be available late in the project. This stepwise approach makes it inevitable to devise an algorithm that is capable off keeping some of its knowledge from previous steps and adapt it for the next level. That makes it unnecessary to learn from scratch for each new component and lowers development and production costs. The idea of transfer learning was already described in Section 5.3: adapting a pre- trained algorithm to a new domain to save time and training effort, especially when there is not enough data available in the new domain. The common and straightforward approach to do so for neural networks is called fine-tuning. The following four steps sketch the approach in a simplified manner. 1. Load network topology and initialise it with pre-trained weights 2. Cut off last layer - the output layer 3. Add new output layer that fits the new problem 4. Train the last network again and keep all old layers in their original state, do not update their weights This approach, originally described by Hinton et al. [5] has the advantage that the training converges and it takes much less time to train compared to training from scratch. In our case, we use simulated data for pre-training and the real data of one type of component for fine-tuning. For the more sophisticated parts, the network can be adapted again. An approach that could be an even better fit to our multi- component/task problem are progressive neural networks [20] because their topology is especially made for transfer learning from one to multiple tasks. The principle works as follows and is shown in Fig. 5.2: 1. Train a neural network with an arbitrary topology (CNN, RNN, ...); this is the first column. 2. Add a second network/column that is trained for another task alongside the first column. 68 S. Stieber

Fig. 5.2: Progressive neural network with three columns from [20].

3. Add adapter connections from the layers of the first to the second column (denoted (a) in Fig. 5.2). 4. Column one is disabled for training. 5. Column two uses features from column one to solve the new task. This approach has some great advantages: it circumvents catastrophic forgetting [1], an unwanted behaviour that happens to fine-tuned networks very often: once a new task is learned, the algorithm performs poorly in the original task. Another interesting feature for our purpose is that new tasks can be added easily: one new column per task. Helpfully, new tasks do not have to contain as many parameters as the previous ones and thus need less data. For our project a good first start is to train the first column for simulated data for the simple component. The second column could be the real data for the simple component. This learning from previous stages continues when switching to the chicane component: it is very desirable to keep the training effort low when transferring knowledge. The disadvantage of these networks is the massive amount of parameters they utilise. For every new task, a new column is added, which lets the number of weights grow enormously when training many tasks. In our project, we do not have to meet hard storage constraints but keeping an eye on the dimensions of a network is always reasonable. In the explanation above, one detail was skipped: the adapter connections between the layers of different columns. These connections are single layer Multi Layer Per- ceptrons (MLPs), which means they have weights and activation functions and thus 5 Transfer Learning for Optimisation of CFRP 69 can be disabled. As the number of columns grows, the number of adapters and their related parameters grows exponentially. Since the number of parameters in a progressive neural network is high anyway, disabling certain connections between layers helps to keep the size of the network in check. For the different components, we obtain simulated data from flow front simulations. As described in Section 5.3, the preferred simulation software is PAM RTM by ESI. Flow front simulations have certain similarities to videos: the single images are 2D if the component is not too complicated and they have a time series nature. Colors indicate different stages of wetting and/or polymerisation. One obvious path to handle this type of data is to treat it similar to video data which opens the possibility to adapt approaches for video analysis that have proven successful: previous studies include e.g., video classification as shown by Karpathy et al. [8] and Ng et al. [27]. Another idea based on the video path is to use image segmentation methods [13] to pre-train a network in a first step to let it gain knowledge on segmenting flow fronts and the optimal distribution of resin and curing. In a second step, that network is retrained for regression, i.e. inferring one value for the quality of the component. Convolutional Neural Networks (CNNs) are neural networks that work very well for image segmentation and overall analysis of single images. If we follow the image path, a mixture of Convolutional Neural Network (CNN)s is the network topology we choose, no matter if we fine-tune a network or use several networks together in a progressive neural network. For CNNs, the hyperparameters to tune are: the overall topology, which can be a residual network, or made of inception modules or simpler topologies or another architecture. The next parameter is the number of layers and neurons. For training, the learning rate and the optimiser are interesting parameters to tune. We will start with proposed values and start optimising training as soon as progress is visible, that means the training is working. The idea of using raw sensor data input would make it necessary to select of a different kind of topology that is well suited for that data because it is uncertain if raw data has the same properties as images: repeating patterns at certain areas, areas of higher interest than others and likewise. Since we do not know yet if we can use the raw data for learning, this path of raw data is not described further here. Despite the fact that deep neural networks have proven to work for many different kinds of problems, we will consider other types of algorithms for our approach. Sorg [23] showed that a method for constructing decision trees, which was invented a long time ago, CHAID [9], was more suitable for his optimisation problem of the process mining the production of a carbon car roof. When it comes to the acceptance of an algorithm within production and management, a very important aspect of an algorithm is its interpretability. Decision trees are easily understandable whilst neural networks are black boxes with next to no interpretability. If the performance figures of these algorithms are similar and even if the decision tree is a little weaker than the neural network, chances are that the overall decision rules in favor of the understandable decision trees. Decision trees carry another interesting feature: they do not need as much training data as deep neural networks. Opting for them, or any other data frugal 70 S. Stieber algorithm would have the advantage of setting aside using simulated data, as well as transfer learning and just use the real data collected by the sensors. That would be a much simpler approach. For our goals (the detailed construction of a flow front, curing, etc.), these algorithms are likely not capable enough, but we will give them a try as well. One other simple approach are shallow neural networks as utilised by Pan- telesis et al. [19]. But even for these networks carrying only one hidden layer with a handful of neurons, simulated data was used for training. This reduces the hope that only real data will help to successfully train any kind of learning algorithm. The real data is coming from sensors within the press: ultrasonic, electric, pressure and temperature sensors, which are also shown in Fig. 5.1. Deriving the flow front and the curing and all other target values from this data is another challenge, which will be tackled with the help of experts in material science. After this first step of predicting quality of a finished component from sensor input, optimising the quality of components over the course of time is the next level of complexity. With the acquired quality inference method, a Reinforcement Learn- ing (RL) approach to increase the quality of the component will be developed and applied. The parameters to change, e.g., the actions for RL, are the pressure of resin, the pressure of the press and the temperature of the press. Ultimately, we will optimise our algorithm to recognise possible defects in the component in the press and to consequently take countermeasures such as increasing the pressure of the press or leaving the component in the press for a longer time. To obtain a baseline to compare to, approximately 100 units of the simple sample components will be produced. Their quality will be tested and they will therefore be destroyed. This quality assessment of 100 simple components will work as our labels and we will split this set into training, test, and validation data sets. We will then compare the outcome our algorithm produces with this real data. For the more complex component, not as many units will be produced. Nonetheless, the real data will also be used for training and testing, but at a smaller scale.

5.5 Outlook

We suggested transfer learning from simulated data as a tool to make DL usable for domains with small datasets. The optimisation of the production of CFRP is the use case our project is dedicated to. We showed the expected challenges and the proposed methodology for that project. After a training for PAM Composites, enough data for initial neural network trainings have to be produced and the first trainings have to be carried out. These two steps, gathering data and training, including tweaking of parameters have to be repeated until the results satisfy our demands. When the first batch of sample components and related quality data is completed, we can start our transfer learning effort. Depending on the success of these attempts 5 Transfer Learning for Optimisation of CFRP 71 on transferring knowledge, the following steps include, first, to incrementally optimise the quality of components with RL over time, and second, to interfere in the production process, if necessary. At last, adapting knowledge from the simple to the complicated component has to be accomplished. Our part in this project is comprised of all data science and Machine Learning (ML) tasks. At the moment, especially the work on transfer learning and the combination of various learning methods are scientific challenges. Others will follow during the lifetime of the project. The research of these problems will be a major part of my thesis.

References

1. French: Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3(4), 128–135 (apr 1999) 2. Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP) (6), 6645–6649 (2013) 3. Grossing,¨ H., Stadlmajer, N., Fauster, E., Fleischmann, M., Schledjewski, R.: Flow front advancement during composite processing: predictions from numerical filling simulation tools in comparison with real-world experiments. Polymer Composites 37(9), 2782– 2793 (sep 2016) 4. Gudemann,¨ M., Nafz, F., Ortmeier, F., Seebach, H., Reif, W.: A specification and construction paradigm for organic computing systems. In: Proceedings - 2nd IEEE Inter- national Conference on Self-Adaptive and Self-Organizing Systems, SASO 2008. pp. 233–242 (2008) 5. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006) 6. Ingrid Bonde Akerlind, Parisa Bastani, Irene Berry, Kandarp Bhatt, Alice Chao, E.C., Valerie Karplus, David Keith, Michael Khusid, Eriko Nishimura, S.Z.: On the Road toward 2050: Potential for Substantial Reductions in Light-Duty Vehicle Energy Use and Greenhouse Gas Emissions John. Massachusetts Institute of Technology (2015) 7. Kalafat, S., Sause, M.G.: Localization of Acoustic Emission Sources in Fiber Compos- ites Using Artificial Neural Networks. 31st Conference of the European Working Group on Acoustic Emission pp. 1–8 (2014) 8. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-Scale Video Classification with Convolutional Neural Networks. 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1725–1732 (2014) 9. Kass, G.V.: An Exploratory Technique for Investigating Large Quantities of Categorical Data. Applied Statistics 29(2), 119 (1980) 10. Kostka, P., Holeczek, K., Filippatos, A., Langkamp, A., Hufenbach, W.: In situ integrity assessment of a smart structure based on the local material damping. Journal of Intelli- gent Material Systems and Structures 24(3), 299–309 (2013) 11. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet Classification with Deep Convo- lutional Neural Networks. Advances In Neural Information Processing Systems pp. 1–9 (2012) 72 S. Stieber

12. Larrosa, C., Lonkar, K., Chang, F.K.: In situ damage classification for composite lam- inates using Gaussian discriminant analysis. Structural Health Monitoring 13(2), 190– 204 (2014) 13. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 07-12-June, pp. 3431–3440. IEEE (jun 2015) 14. Lv, Y., Duan, Y., Kang, W., Li, Z., Wang, F.Y.: Traffic Flow Prediction With Big Data: A Deep Learning Approach. IEEE Transactions on Intelligent Transportation Systems 16(2), 865–873 (2014) 15. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human- level control through deep reinforcement learning. Nature 518(7540), 529–533 (feb 2015) 16. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 17. Nafz, F., Seebach, H., Steghofer,¨ J.p., Anders, G., Reif, W.: Constraining Self- organisation Through Corridors of Correct Behaviour: The Restore Invariant Approach. In: Organic Computing A Paradigm Shift for Complex Systems, pp. 79–93. Springer Basel, Basel (2011) 18. Pantelelis, N., Vrouvakis, T., Spentzas, K.: Cure cycle design for composite materials using computer simulation and optimisation tools (April) (2003) 19. Pantelesis, N., Bistekos, E.: Process monitoring and control for the production of CFRP components. SAMPE Conference pp. 5–9 (2012) 20. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive Neural Networks. arXiv (2016), http://arxiv. org/abs/1606.04671 21. Rusu, A.A., Vecerik, M., Rothorl,¨ T., Heess, N., Pascanu, R., Hadsell, R.: Sim-to-Real Robot Learning from Pixels with Progressive Nets. In: 1st Conference on Robot Learn- ing (CoRL 2017). pp. 1–9 (2016) 22. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., Hassabis, D.: Mastering the game of Go without human knowledge. Nature 550(7676), 354–359 (oct 2017) 23. Sorg, C.: Data Mining als Methode zur Industrialisierung und Qualifizierung neuer Fer- tigungsprozesse fur¨ CFK-Bauteile in automobiler Großserienproduktion. Dissertation, Technische Universitat¨ Munchen¨ (2014) 24. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 25. Weiss, K., Khoshgoftaar, T.M., Wang, D.D.: A survey of transfer learning, vol. 3. Springer International Publishing (2016) 26. Wu, J., Yildirim, I., Lim, J., Freeman, W., Tenenbaum, J.: Galileo : Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning. Advances in Neural Information Processing Systems 28 (NIPS 2015) pp. 1–9 (2015) 27. Yue-Hei Ng, Joe and Hausknecht, Matthew and Vijayanarasimhan, Sudheendra and Vinyals, Oriol and Monga, Rajat and Toderici, G.: Beyond Short Snippets : Deep Net- works for Video Classification. Proceedings of the IEEE conference on computer vision and pattern recognition pp. 4694—-4702 (2015) 5 Transfer Learning for Optimisation of CFRP 73

28. Zhang, J., Pantelelis, N.: Modelling and optimisation control of polymer composite moulding processes using bootstrap aggregated neural network models. 2011 Interna- tional Conference on Electric Information and Control Engineering, ICEICE 2011 - Proceedings 1(2), 0–3 (2011)

Transfer Learning in the Field of Renewable Energies A Transfer Learning Framework Providing Power Forecasts Throughout the Lifecycle of Wind Farms After Initial Connection to the Electrical Grid

Jens Schreiber

Intelligent Embedded Systems Lab, University of Kassel, Kassel, Germany [email protected]

Abstract. In recent years, transfer learning gained particular interest in the field of vision and natural language processing. In the research field of vision, e.g., deep neural networks and transfer learning techniques achieve almost perfect classification scores within minutes. Nonetheless, these techniques are not yet widely applied in other domains. Therefore, this article identifies critical challenges and shows potential solutions for power forecasts in the field of renewable energies. It proposes a framework utilising transfer learning techniques in wind power forecasts with limited or no historical data. On the one hand, this allows evaluating the applicability of transfer learning in the field of renewable energy. On the other hand, by developing automatic procedures, we assure that the proposed methods provide a framework that applies to domains in organic computing as well.

6.1 Introduction

In recent years, transfer learning (TL) [22] gained particular interest in the field of vision and natural language processing (NLP) [17]. In the research field of vision, e.g., so-called deep neural networks [6] are trained on large amounts of data to obtain a computer model that allows classification of almost arbitrary categories in pictures. By training deep neural networks on large datasets, such as ImageNet1, the model learns a high-dimensional representation of the data. Transferring generic representation to other more specific domains, e.g., the classification of cats and dogs is called domain adaption or transfer learning. More specifically, the field of domain adaption comprises transferring a model or data from a source domain to a target domain, where domain refers to the input data of the model. Typically, we distin- guish between source task, the classification in the ImageNet dataset based on the

1 http://www.image-net.org/ (last access: 02/10/2018) 76 J. Schreiber input data; and the target task, the classification of cats and dogs in pictures. TL is especially interesting when there is little or no data in the target domain and thereby helps in improving performance [12, 21] by making use of the knowledge from the source task and domain. Moreover, this approach allows reducing computational cost by adopting an already well-performing model on a new problem. Despite all these benefits, TL methods applied to vision and NLP tasks are not yet often used in the area of renewable power forecasts, even though they have various applications. Traditionally, a physical model was used, in renewable power forecasts, to predict so-called day-ahead estimates of the expected power generation to integrate these volatile energy resources into the electrical grid. The physical wind power model uses rotor area as well as wind speed and air density from numerical weather prediction (NWP) as inputs to forecast the expected power. However, in various articles, it has been shown that Machine Learning (ML) techniques, such as neural networks, have superior forecast quality compared to the physical model[3]. This improvement is, e.g., due to the capability of neural networks to learn the characteristics of a wind farm’s location based on the non-linear relationship of the NWP input and the histor- ically measured wind power generation. Respectively, the improved forecast quality allows decision makers to devise more confident decisions compared to the physical model. However, traditional ML techniques, such as linear regression, support vector regression, or random forests, are not applicable if there exist limited or no historical wind power data. E.g., consider the situation where a new wind farm is constructed or a night cutoff of the turbines is implemented: In these situations limited or no historical data of the generated power is available to train an ML model. So instead we need to fall back to the physical model that requires no training data. This fall back is problematic because decision makers require the excellent forecast performance from ML models to devise a confident justification for the electrical grid at all times. Fortunately, TL techniques allow using ML methods even with little to no data by inferring knowledge from wind farms where considerable knowledge exists. Correspondingly, the thesis aims to develop a framework that allows utilising TL in wind power forecasts with limited or no historical data in the lifecycle of a wind farm. Therefore, the thesis intends to evaluate the applicability of TL in the field of renewable energy by answering the following questions: 1. How can historical source domain data and forecast models transfer to the target domain? 2. What historical source domain data and forecast model is the most suitable to transfer? 3. How to integrate an increasing amount of historical data into the model for the new wind farm? By developing automatic procedures that configure the most suitable data selection themselves, the proposed methods provide a framework that will apply to domains in Organic Computing (OC) as well [11, 20]. Furthermore, self-improvement 6 Transfer Learning in the Field of Renewable Energies 77 techniques based on the configured and selected data allow a continuous self- optimisation framework based on the pool of information. Moreover, the proposed methods are probably applicable to other domains as well. In traffic scenarios, TL can be used to continuously improve the forecasts of pedestrian movements based on various sensor data such as cameras, gyroscope, and GPS. Moreover, models developed for pedestrians are, e.g., transferred automatically for bicyclist or classification tasks. The remainder of this article is structured as follows. First, we summarise related work in Section 6.2. Then we detail the research proposal in Section 6.3. Based on the research proposal we describe the current progress in Section 6.4.

6.2 Related Work

In this section related work relevant to the research proposal in Section 6.3 is detailed. First, Section 6.2.1 gives a general overview of the current research in TL. Afterward, we make the connection to current TL techniques applied in renewable energies in Section 6.2.2 and introduce papers relevant to the research proposal.

6.2.1 Transfer Learning

Transfer learning is the area of research concerning methods allowing the adaption of knowledge from one domain to another. In particular, it is about methods that allow a domain adaption from a source domain DS to a target domain DT . A domain is defined by D = {X ,P(X)}, where X is the feature space and P(X) is the marginal distribution with X ∈ X . The task of a domain relates the NWP input to the generated power and is defined with T = {Y , f (·)}, where the function f (·) relates the input features X to the generated power Y, with Y ∈ Y . Equally to the domain distinction, we differentiate between the source task TS and the target task TT . Other formulas are also differentiated by the subscript S for the source and T for the target. The above definitions allow describing various domain adaption problems, e.g., inductive learning, which tackles the difference between DS and DT . A comprehensive overview of definitions with examples is given in Table 6.1 and Table 6.2. Based on these tables, the following list provides some of the most important concepts towards solutions of the research questions: • Parameter transfer finds shared parameters between the source and the target domain. These shared parameters are used to transfer knowledge from DS to DT [21, 12]. One potential method is an multi-task-learning (MTL) approach. In MTL multiple tasks are learned at once. Consider an ML model, which learns the prediction of 3 offshore wind farms at once. This learning process will allow finding shared parameters that apply to other similar offshore farms. One 78 J. Schreiber

common MTL approach is hard parameter sharing. In hard parameter sharing all hidden layers of a neural network, are shared across all tasks. Each task has several output layers[14, 15]. Another method is to use finetuning of a neural network. First, in finetuning, a deep neural network is trained on a large set of wind farms. Second, the model’s weights are slightly modified (finetuned) to be applicable in the target domain. • Feature representation transfer intends to find a representation of the NWP features minimising the differences between the source domains and the target domains [12]. An autoencoder [9] can achieve this generic representation. An autoencoder is an unsupervised learning strategy that first compresses NWP features (encode) and then restores the original features from the compressed representation (decode) [6]. This encoding and decoding process allows the autoencoder to learn a common feature representation between the source and the target domain. Other approaches also use an MTL approach, where a so-called embedding layer is applied to determine a generic representation of the source domain. The learned generic source representation is used to finetune for the additional target task [10]. • Instance transfer makes use of historical DS, YS and some historical target task data YT . By weighting the little historical data YS, instance transfer allows making the best use of both datasets concerning the forecast quality in the target domain. One method is the self-training approach [17]. First, the self-training approach artificially creates target task data, by a model trained on the source dataset. Second, training a new model on the target domain and the artificially created target response. By only using artificial target data with high confidence for the prediction, it is guaranteed that the data is re-weighted in a way, that just data points contribute to the model that are relevant in the target domain or one could say are transferable.

Table 6.1: Terminologies deﬁnitions of transfer learning. ”-” indicates that the deﬁnition does not make any assumption about the domain or the task.

Domain Task Inductive Learning [21, 12] - TS TT Transduchtive Learning [21, 12] DS DT TS = TT Homogenous Learning [22] XS = XT - Heterogeneous Learning [22] XS XT -

6.2.2 Wind Power Forecasting

In this section, we summarise current research on transfer learning in renewable energies. Further on an ensemble technique is introduced that is applicable in the context of transfer learning. In [9] the authors use transfer learning for wind speed predictions. The authors train a so-called MTL deep denoising autoencoder on four wind farms simultaneously. This 6 Transfer Learning in the Field of Renewable Energies 79

Table 6.2: To refine the terminology and combine different definitions from [21, 12], and [22] the following table gives detailed information about differences in definitions and provides examples in the context of renewable energies.

Domain Task Example Same weather model and input features are used in the source and target domain, Homogenous e.g., wind speed, humidity and wind Inductive XS = XT T T S T direction. The source task is adapted to Learning classify iced wind turbines in the target task , instead of forecasting the power generation. Source and target domain have a different number of weather features, e.g., temperature Heterogeneous is additionally used in the target domain. Inductive XS XT T T S T The source task is adapted to classify Learning iced wind turbines in the target task, instead of forecasting the power generation. Same weather model and input features are used in the source and target domain, Homogenous e.g., wind speed, humidity and wind direction. XS = XT , Transductive T = T However, two wind farm locations with different P(X ) P(X ) S T Learning S T marginal distributions of the weather. Source and target task both forecast power generation. Two NWP models with related but different features of one wind farm location of the source Heterogeneous and the target domain. The underlying marginal XS XT , Transductive T = T distributions of XS and XT are the same, P(X )=P(X ) S T Learning S T due to the similar location of the wind farm. Source and target task both forecast power generation. model allows them to obtain the best forecast results for short time forecast horizons (10-min, 30-min, and 1-h) with a limited amount of data. The author’s of [13] use a combination of nine autoencoders, deep neural networks, and transfer learning to limit the time required for training. Interestingly, initially, all nine autoencoders are trained based on data from a single wind farm and only fine-tuned for the other wind farms. Even though the autoencoders were initially only trained on a single wind farm, the evaluation results show improved results compared to similar techniques. In [23], the authors, use inductive transfer learning to classify iced wind turbines. First, a neural network is trained on a single wind turbine B, and then it is finetuned on a group of wind turbines A. Afterward, the classifier is tested on a remaining group of 99 wind turbines from group A. The repeated experiment for 100 different combinations, shows an improvement of 14% compared to an oversampling strategy. 80 J. Schreiber

[18] proposes a cluster based multi source domain adaption (MSDA) approach. The approach allows to cluster similar wind data distributions based on historical weather records from multiple sites. These wind distributions allow fusing predictions from multiple source sites, utilising a weighting scheme, for a new wind farm. The cluster- based MSDA achieved an improvement of 20.63% compared to a traditional MSDA approach. [4, 5] present the coopetitive soft-gating ensemble (CSGE) in the context of renewable power forecasts. It comprises a hierarchical two-stage ensemble prediction system and weights the ensemble member’s predictions based on three aspects, namely global, local, and time-dependent performance. [2] shows that it is utilisable with an arbitrary cost function. The results in [2] allow to implement the CSGE in the context of TL with a suitable cost function, as detailed later on. In summary, to the best of our knowledge, there is yet little research towards an evaluation on the applicability of TL in the ﬁeld of renewable energy, that allows reutilising data and models from other wind farms throughout the whole lifecycle from its ﬁrst operation on the electrical grid.

6.3 Research Proposal

The following sections provide details of the research proposal based on the previously mentioned research questions. By solving three import challenges, we evaluate the applicability of TL in the field of renewable energy. Solving these challenges allow reutilising data and models from other wind farms throughout the whole lifecycle from a wind farm’s first operation on the electrical grid answering question one and two. Besides, the developed methods are capable to gradually integrate an increasing amount of data answering the research question three. To develop these methods and provide a solution we identified the three challenges that are relevant to tackle from an ML perspective as well for renewable energies. Challenge 1 in Section 6.3.1 discusses problems and solutions that arise when there is no historical power data about a wind farm showing strategies to select models and data from other wind farms in the absence of no historical power data; answering the research question one and two. Challenge 2 in Section 6.3.2 discusses problems that arise when there is limited historical power data about a wind farm; answering the research question one and two. However, it also shows the potential improvements from the additional historical data. Challenge 3 in Section 6.3.3 deals with problems arising from the addition of larger historical data sets. In particular, it looks at how a change in input data compared to old data affects estimates of the wind power forecast. Also, it considers what volume of historical data is required so that traditional ML instead of TL approaches can be applied. Respectively this section answers all three research question with a strong focus on the third question. 6 Transfer Learning in the Field of Renewable Energies 81

6.3.1 Challenge 1: Providing Day-Ahead Forecasts of the Expected Power Generation without Historical Power Data

Typically, when there is no historical power data of a new build wind farm, we need to use the physical model to provide a forecast of the expected power generation. The model has the rotor area, forecasted wind speed, and forecasted air density from a NWP as inputs to provide day-ahead estimates of the expected power generation. These estimates of the model are used to devise decisions regarding costs and risks in the electrical grid. However, even though there is no historical power data for a new site, typically information from the source in the form of historical DS knowledge and models TS from other wind parks are available. This knowledge DS and TS allows providing a more sophisticated forecast in the target domain compared to the physical model. This information of data and models is adapted from the source domain to the target domain with TL techniques. Further on we typically can extract historical NWP DT about the location of the new wind farm and has information about the location’s terrain (onshore, offshore, forest, farmland, and central mountain zone). All these pieces of information can be used to select relevant models and historical data from other wind farms as detailed later on. Respectively, in this challenge the following difﬁculties arise:

1. Historical data XS and YS from other wind farms is probably distorted towards the specifics of its original location. E.g., turbulence caused by surrounding trees influence an onshore wind farm located in the forest. These turbulences are different compared to effects occurring in offshore wind farms [16]. 2. Similar to the historical data, forecast models TS are probably biased towards specifics of a particular wind farm [12]. Further on, the parameters of the machine learning models are biased towards the data of the original wind farm and may be hard to adapt for the new wind farm. To cope with these challenges, we aim to solve it through a two-step approach as shown in Figure 6.1. Step 1 automatically configures a suitable selection of historical data and forecast models that are most similar to the new wind farm. This self-configuration can be made by analysing influences in forecast models and derive respective rules that allow identification based on the location or historical NWP data. The location-based similarity identification is a pre-selection strategy for the second step. It minimises the risk of distortion towards the specifics of the source domain Step 2 derives a model based on pre-selected historical data and models using TL approaches. One naive approach for Step 2 is to artificially create historical wind power data for the new wind farm. The pre-selected forecast model can do this with NWP from the new wind farm’s location as input. Afterwards, we can train a machine learning algorithm for the new wind farm with the historical NWP data, and the artificially created power data. Another method is to create a high-dimensional representation of the NWP data for the new wind farm using an autoencoder and then use a universal 82 J. Schreiber forecast model trained on the historical data of similar wind farms. These two semi- supervised approaches will provide a self-optimisation scheme for new wind farms.

Fig. 6.1: Scheme depicting the procedure to use domain adaption for a new wind farm. First, comparable wind farms are identiﬁed through similar weather conditions or terrain. Second, model and historical data of similar parks are adapted to be applicable for the new wind farm.

6.3.2 Challenge 2: Providing Day-Ahead Forecasts of the Expected Power Generation with little Historical Power Data

Similar to the challenge from Section 6.3.1, we aim at developing an ML model that allows day-ahead forecasts. In particular, we are interested in the scenario with limited YT of a new wind farm. This historical data includes observations from, e.g., the first month after the beginning of the wind farm’s first operation. Ideally, this data allows developing a model that improves the forecast quality over the physical model as well as other models developed in Section 6.3.1. Even though the problem is similar to the one in Section 6.3.1, the additional YT has different challenges as well as potential solutions: 1. One problem is when the new wind farm has a characteristic curve that is not part of the historical wind farm pool. This mismatch can happen due to a false pre- selection strategy of the similar wind farms for example due to a yet unknown wind turbine type or topology. 2. The few historical data of the first month might be biased towards specific weather conditions only applicable to this month. Similar problems occur in case initial maintenances at the wind farm distort the first month’s data. What the described challenges above have in common is that the model could learn a relationship between input and output that is specific to the historical data. Con- sequently, the model will not generalise well for the next’s month data. In the worst scenario, the data causes negative transfer, so that the new model is worse than the one from similar wind farms or the one from Section 6.3.1. 6 Transfer Learning in the Field of Renewable Energies 83

Similar to Section 6.3.1, we first use the pre-selection strategy to configure the pool of relevant wind farms. Based on this pool we propose three potential solutions for Step 2 in Figure 6.1. The first one is to use an ensemble technique such as the coopetitive soft-gating ensemble (CSGE) that provides a self-configuration and self- optimisation scheme for Step 2 of the challenge. The CSGE has a so-called local and global weighting scheme. The global weighting scheme provides an assessment strategy to evaluate and weight the pre-selected models on the historical data of the first month. In a sense, here, this can be used to derive a weighting on how similar the models are to the new wind farm even with few historical power data. At the same time, regardless of the global weighting, the individual models provide forecasts for data not yet measured at this location to avoid negative transfer. The local weighting scheme allows deriving a weighting based on similar weather situations. This local weighting probably further mitigates the limited amount of historical power data. Further on, by adding a model to the ensemble that is trained only on the historical data of the new farm, we can avoid the first challenge, because its performance will be taken into account by the global and the local weighting scheme for the unknown characteristic curve. The second approach is to use a combination of finetuning and the so-called self- training approach. Here, first, a generic model based on the pool of all similar wind farms is trained. This model is then finetuned based on the historical data for the first month. The resulting model is then used to create forecasts of the power for historical NWPs of the new wind farm. The artificially created historical wind power allows to further train the initial finetuned model based on confident prediction (e.g., 90%) until a specified stopping criterion is met. The combination of artificially creating historical data and retraining of the model is called self-training. By training the model on the artificial data, the self-training approach improves the model for other weather conditions. Further on, by just selecting data with high confidence the self- training avoids negative transfer for unknown data. The third approach is to use an MTL strategy based on hard-parameter sharing. We first learn a model based on the pre-selected pool of similar wind farms. In the multi- task approach, all wind farms are trained at the same time. This simultaneous training allows the algorithm to derive a universal representation of the NWP data. Once, the model is trained we add the task for the new wind farm and finetune the model. Due to the generic nature of the NWP representation, only a small amount of historical data is needed to adapt the weights for the new task. Because of a large amount of data for the initial training, typically the generic representation allows deriving sophisticated models even with few historical data and provides a suitable self- optimisation framework. Further, by the adaption strategy based on the one month of training data, we assure that the algorithm learns the specifics of the new wind farm compared to the universal representation. The combination of historical data specific to the wind farm and the generic representation avoids the negative transfer by only learning the relevant data valid to the farm. 84 J. Schreiber

6.3.3 Challenge 3: Providing Day-Ahead Forecasts of the Expected Power Generation with Increasing Historical Data and Lifetime

While in Section 6.3.2 there was little historical data, with increasing lifetime of a wind farm more and more data YT is collected. Based on the model created in Section 6.3.2 we now need to integrate this data into the algorithms gradually. The continuous integration of new historical data YT increases the forecast performance. However, a change in the data compared to past data affects estimates of the wind power forecast. This change in data can be related to a newly integrated night shut- off, maintenance, or the implementation of a different weather model. Respectively, this challenge has the following difficulties: 1. How to detect and adapt the model for unknown situations such as maintenance? 2. How to minimise the training effort concerning computational costs for conceptual drifts in the data, e.g., from a new weather model? Importantly to note, the selection from the pool of similar wind farms is here potentially different to the other challenges. This difference is because previously we had little information about the farm and were only considering data from similar locations and characteristics. However, now we are interested in similar situations. In the case of a night shut-off, the data is identical for all wind farms that implement it. In all implementations, the generated power will be zero within a certain time span. Recognising such changes is typically solved with anomaly or novelty detection [8, 7]. Afterwards, we can use a clustering algorithm such as k-nearest neighbours to find similar training data from other wind farms. After finding the related pieces of training data, we are again able to use finetuning and self-training to adapt the original model to the novel situations as described in Section 6.3.2. By using finetuning, we can minimise computational costs compared to traditional ML approaches. An iterative process of the CSGE gives a more automatic approach. Consider we have a CSGE, where we combine models of all wind farms. With an increasing amount of data for the new wind farm, the individual model’s performance will increase and respectively its weighting compared to other wind farms. However, now consider the installation of night shut-off. Once, we retrain the CSGE with data containing the night shut-off, the global performance of the novel wind farm model will decrease while the models with knowledge about night shut-off will increase. This retraining of the CSEGE allows adapting to novelties automatically. Additionally, one also needs to finetune the new wind farm model based on the new data, to assure that the model learns to predict night shut-offs or similar scenarios over time. An MTL approach tackles the scenario with a new NWP model. Consider a neural network, where multiple NWP models are the input for the MTL model. Further, the model is trained so that the power generation forecast is independent of a specific weather models input. This approach will force the algorithm to learn a generic representation of the NWP models. Respectively, allowing to apply domain adaption techniques for a new NWP model. 6 Transfer Learning in the Field of Renewable Energies 85

This idea can be further extended with a spatial abstraction layer, see Figure 6.2, to what we call multi-cross-learning. Initially, during training, the model learns with data from l weather models and k wind farms at the same time. The simultaneous training causes the model to determine a generic representation of the NWP features on the one hand and of the spatial relation of the wind farms on the other side. The resulting model allows tackling various problems at once. In case we are faced with novelty for wind farm i, domain adaption of the spatial abstraction will enable to use models from other wind farms. Further, this allows integrating new NWP models easily. The generic abstraction of the NWP allows limiting the training to a transformation between the new NWP model and the NWP abstraction. Finally, this approach can be considered as an extension of the self-optimisation scheme from the traditional MTL approach by providing additional layers of abstraction.

Fig. 6.2: Multi-task-learning, which serves to transfer knowledge about numerical weather models as well as to model the dependencies between individual wind parks and their prediction models. Multiple NWP models are the input for the MTL model. Further, the model is trained so that the power generation forecast is independent of a speciﬁc weather model’s input.

6.4 Progress Description and Summary

In this section, we summarise our preliminary work and outline the research proposal. Preliminary work in [16] provides a method that analyses the influence of so- called probabilistic forecasts method. The results of this method will be used later on to design the pre-selection strategy as described in Section 6.3.1 and 6.3.2. There- fore in [16], a hybrid feature selection strategy is implemented, and influences are analysed with the so-called sensitivity analysis. The results show that, as expected, wind speed at 100m altitude is the essential feature in all evaluated terrains and ML models. Similar, in most cases wind speed at 10m height has the second most significant influence. If selected, air pressure and the variability of wind speed at 100m altitude have about the third highest impact for all evaluated scenarios. In [2], we proposed the CSGE for general machine learning tasks and interwoven systems [1, 19]. This paper shows that the CSGE can be optimised according to ar- 86 J. Schreiber bitrary loss functions making it accessible for a broader range of problems and providing a self-improving scheme based on previously seen data. This self-improving scheme can be applied to the self-configuration and self-optimisation scenarios as proposed in Section 6.3.2 and 6.3.3. Moreover, we showed the applicability and easy interpretability of the approach for synthetic datasets as well as real-world data sets. For the real-world datasets, we showed that our CSGE approach reaches state-of- the-art performance compared to other ensembles methods for both classification and regression tasks. Conclusively, in this article , we proposed various techniques that allow utilisation of transfer learning techniques in the field of renewable power forecasts. The proposal is the first step toward a broader and automated use of transfer learning in other domains besides NLP and vision. We are currently in the process of investigating the multi-cross task learning approach and the superior forecast quality of transfer learning over the physical model. Initial findings show the applicability and motivate further research in this area.

References

1. Bellman, K.L., Tomforde, S., Wurtz,¨ R.P.: Interwoven systems: Self-improving systems integration. In: Eighth IEEE International Conference on Self-Adaptive and Self- Organizing Systems Workshops, SASOW 2014, London, United Kingdom, September 8-12, 2014. pp. 123–127 (2014) 2. Deist, S., Bieshaar, M., Schreiber, J., Gensler, A., Sick, B.: Coopetitive Soft Gating Ensemble. In: Workshop on Self-Improving System Integration at IEEEICAC/ SASO (2018) 3. Gensler, A.: Wind power ensemble forecasting. Ph.D. thesis, Univerisity of Kasel (2018) 4. Gensler, A., Sick, B.: Forecasting wind power - an ensemble technique with gradual coopetitive weighting based on weather situation. In: IJCNN. pp. 4976–4984. Vancou- ver, BC (2016) 5. Gensler, A., Sick, B.: A multi-scheme ensemble using coopetitive soft gating with application to power forecasting for renewable energy generation. CoRR arXiv:1803.06344, 1–22 (2018) 6. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016) 7. Gruhl, C., Sick, B., Wacker, A., Tomforde, S., Hahner,¨ J.: A Building Block for Aware- ness in Technical Systems: Online Novelty Detection and Reaction With an Application in Intrusion Detection. In: Proceedings of the IEEE 7th International Conference on Awareness Science and Technology (iCAST 2015), held Sp. 22-24, 2015, Qinhuangdao, China. pp. 194 – 200. IEEE (2015) 8. Gruhl, C., Sick, B.: Novelty detection with candies: a holistic technique based on probabilistic models. International Journal of Machine Learning and Cybernetics 9(6), 927– 945 (2016), https://doi.org/10.1007/s13042-016-0618-8 9. Hu, Q., Zhang, R., Zhou, Y.: Transfer learning for short-term wind speed prediction with deep neural networks. Renewable Energy 85, 83–95 (2016) 10. Long, M., Cao, Y., Wang, J., Jordan, M.I.: Learning Transferable Features with Deep Adaptation Networks. CoRR arXiv:1502.02791, 1–9 (2015) 6 Transfer Learning in the Field of Renewable Energies 87

11. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing - Technical Systems for Survival in the Real World. Birkhauser¨ (2017), https://doi.org/10.1007/978-3-319-68477-2 12. Pan, S.J., Yang, Q.: A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345–1359 (2010), http://ieeexplore.ieee.org/document/ 5288526/ 13. Qureshi, A.S., Khan, A., Zameer, A., Usman, A.: Wind power prediction using deep neural network based meta regression and transfer learning. Applied Soft Computing 58, 742–755 (2017) 14. Ruder, S.: An Overview of Multi-Task Learning in Deep Neural Networks. CoRR arXiv:1706.05098, 1–15 (2017), http://arxiv.org/abs/1706.05098 15. Ruder, S., Bingel, J., Augenstein, I., Søgaard, A.: Learning what to share between loosely related tasks. CoRR arXiv:1705.08142, 1–12 (2017), http://arxiv.org/abs/1705. 08142 16. Schreiber, J., Sick, B.: Quantifying the Inﬂuences on Probabilistic Wind Power Fore- casts. In: The 3rd International Conference on Power and Renewable Energy (accepted 2018) 17. Søgaard, A.: Semi-Supervised Learning and Domain Adaptation in Natural Language Processing. Synthesis Lectures on Human Language Technologies 6(2), 1–103 (2013), http://www.morganclaypool.com/doi/abs/10.2200/S00497ED1V01Y201304HLT021 18. Tasnim, S., Rahman, A., Oo, A.M.T., Haque, M.E.: Wind power prediction in new stations based on knowledge of existing Stations: A cluster based multi source domain adaptation approach. Knowledge-Based Systems 145, 15–24 (2018) 19. Tomforde, S., Hahner,¨ J., Sick, B.: Interwoven Systems. Informatik-Spektrum 37(5), 483–487 (2014), http://dx.doi.org/10.1007/s00287-014-0827-z, Aktuelles Schlagwort 20. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 21. Torrey, L., Shavlik, J.: Transfer Learning. In: Handbook of Research on Machine Learn- ing Applications, p. 834 (2009) 22. Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. Journal of Big Data 3(1), 40 (2016), http://journalofbigdata.springeropen.com/articles/10.1186/ s40537-016-0043-6 23. Zhang, C., Bin, J., Liu, Z.: Wind turbine ice assessment through inductive transfer learning. In: 2018 IEEE International Instrumentation and Measurement Technology Confer- ence (I2MTC). pp. 1–6. IEEE (2018), https://ieeexplore.ieee.org/document/8409794/

Combining Machine Learning with Blockchain: Beneﬁts, Approaches and Challenges

Wenzel Pilar von Pilchau

Organic Computing Group, Universitat¨ Augsburg, Augsburg, Germany [email protected]

Abstract. Machine Learning (ML) as well as Blockchain are two fields which bear great potential for future research. In this work, four approaches to Blockchain/ML hybrid systems are discussed from an Organic Computing perspective. Smart Contracts are build-in functions of a Blockchain and could be extended with learning components to bring intelligence to the chain. But a Blockchain could also serve as information dealer for ML applications. There- fore it is necessary to develop an efficient feature extraction method which collects the right data as efficient as possible. Also interesting is the use of the Blockchain as a decentralised environment for artificial life. Blockchains and Smart Contracts perform well in their specific domain, but there are limitations. It is not possible for a Smart Contract to obtain information from the outside world, for this purpose so called Oracles are used. The trustworthiness and decentralisation of these could be improved by using ML. The combination of both has not got much attention and some possibilities to achieve this in the context of Organic Computing are discussed in this paper.

Keywords: Machine Learning, Blockchain, Organic Computing, Smart Contracts, Artiﬁcial Life, Feature Extraction, Oracles

7.1 Introduction

Organic Computing (OC) [20] describes decentralised, self-organising systems which establish so-called self-x properties, such as self-learning and self-optimising [28]. In recent years, the term Blockchain [25] as well as the term Machine Learning (ML) [19] has got more and more attention in society in general as well as in the scientific research area. While ML can improve our daily life in every imaginable aspect, the Blockchain is highly used for decentralising processes as well as making them transparent and ensuring immutability of their history. One advantage of using Blockchain is its decentralised character. Another element are smart contracts which are deployed on the chain in form of executable functions that can trigger each other. A complex ecosystem of smart contracts can enable an autonomous self-regulating 90 W. P. von Pilchau system. Nature often archives emergent complex behaviour using decentralised approaches, i.e. ants finding a short path to their source of food. They are not instructed by a single central authority like the queen but interact with each other. A key characteristic of Blockchain is making central authorities unnecessary and in contrast solving problems in a decentralised manner. Both, Machine Learning and Block- chain fit into the paradigm of OC. The contribution of this work is to give on the one hand a short introduction to Machine Learning and Blockchain and on the other hand to search for promising collaborations of these two.

7.1.1 Machine Learning

Machine Learning (ML) addresses the question of how to build computer programmes that improve their performance at some task through experience. It is a multidisciplinary field of research which gets input from results of i. e. artificial intelligence, probability and statistics, computational complexity theory, philosophy and a lot more[19]. Popular representatives are for example reinforcement learning, supervised learning, unsupervised learning and online learning. In Reinforcement Learning an agent interacts with it’s environment to archive a goal [27]. The agent must be able to sense the state it’s is in and also to take actions which affect the environment and so change it’s state. After every performed action a reward is archived, positive or negative, which the agent uses to learn. Over time the the environment is explored and the agent is learning from it’s experience. The goal is to maximise the reward signal and perform optimal in the scenario the agent is in. One of the challenges that arise on the way is the trade-off between exploration and exploitation. To obtain a lot of reward, it is necessary to prefer actions that have already been tried and declared as effective in producing reward. But to discover such actions, the agent has to explore by trying actions it has not selected before [24]. An example for a reinforcement learning agent is a robot in a maze. In contrast supervised learning or “learning with a teacher” describes the process of learning a set of rules from labeled instances (examples in a training set). A classifier is created that can be used to generalise over new, unseen instances. First a dataset is collected and important attributes and features are detected. The classifier is trained on this data and learns to map from unlabeled instances to classes. Common examples for supervised learning are face recognition and image classification [13]. Unsupervised learning or “learning without a teacher” addresses the task of clustering unlabeled data. The goal is to organise the data in a way such that the single items in one cluster are more “similar” to each other, than they are to items in the other clusters. This notion of similarity can be expressed in very different ways, according to the purpose of the study, to domain-specific assumptions and to prior knowledge of the problem [9]. 7 Combining Machine Learning with Blockchain 91

7.1.2 Blockchain

What is this Blockchain and how does it work? In this section, there will be a high level explanation of the technology behind this term. In 2009 an anonymous hacker (or group of hackers) named Satoshi Nakamoto published the Bitcoin white paper [21]. Bitcoin is the world’s first entirely digital and decentral currency. It created a single, universally accessible digital ledger, where the distribution of all existing coins was saved and called it Blockchain. The name derives from the fact that changes can only be made by adding new information to the end. A new added part is called a block and the whole chain is constructed out of these blocks, each of them containing a set of transactions. These transactions refer to previous transactions. For instance, Alice pays Bob a Bitcoin, this transaction appears at the end of the chain and points to the transaction in which Alice was previously paid that coin by Carol, which in turn points to the time before that when Carol was paid the coin by Dave, and so on [23]. One feature of the Blockchain is its decentralisation. There are other approaches like private and hybrid Blockchains, but we will focus on the public ones in this article. The whole chain is replicated on networked computers all over the world and accessible to anyone with a computer and an Internet connection. The participating users are split into different classes: normal users and miners. The miners are important for the whole system and serve several tasks as collecting transaction requests from users, aggregating them, validating them, and adding them to the Blockchain as new blocks. It is important that no one can spend money twice, that is one reason why the miner validates transactions. The other point is to make sure that the sender actually owns enough Bitcoin. Ownership on the Blockchain is determined by a pair of private and public keys and these are used to sign transactions [23]. The main role of miners is, besides the mentioned ones, to ensure the irreversibility of the Blockchain. The method which is used by most Blockchains is called proof of work (PoW). The reason why a mechanism like PoW is needed, is to overcome the Byzantine generals problem [14]. The block interval (the time-interval in which a new block is mined) defines the latency at which new blocks are added to the chain. So once in the block interval, there is one user allowed to create a new block, fill it with waiting transactions, and add it to the Blockchain. To decide which user gets this right, every miner participates with his computing power by solving PoW instances. Bitcoin employs a hash-based PoW which induces finding a nonce value which, when hashed with other parameters from the actual block, results in a smaller value as a target. When a miner found such a nonce and added a new block, the other peers in the network can check if everything is valid, by building the hash of the block [8]. By using the Blockchain technology, it is possible to solve problems without a third party which serves as trustworthy middleman. In the past a number of intermediaries, third parties, where necessary to facilitate transactions. 92 W. P. von Pilchau

7.1.2.1 Smart Contracts

The term smart contract was first mentioned and described by Nick Szabo in 1997 as follows: ”Smart contracts combine protocols with user interfaces to formalise and secure relationships over computer networks” [26]. While Bitcoin’s Blockchain is focused on storing transactions, there arose a new, more powerful, generation. Ethereum [6] implemented a Blockchain with a Turing- complete contracting language which is executed on the Blockchain. It is possible to store programmes called ”smart contracts” in the blocks. The Turing-completeness allows to create complex contracts. Some examples of use are financial derivatives, insurance contracts, and financial exchanges. Moreover, digital services like the rental of digital storage, computational power, and communication bandwidth are also straightforward to implement [22]. If it is possible to execute programmes stored on a Blockchain, there arises a well known issue called the halting problem. This describes the fact, that it is impossible to tell, in general case, whether or not a given programme will ever halt. Malicious users could shut down miners and full nodes by forcing them to enter into an infinite loop. To overcome this issue, Ethereum invented the so called ”gas”, which is a currency to pay for computational steps. If a user wants to execute a smart contract, he has to pay some gas. If the amount of gas was enough to fully execute the programme, then the execution is valid, if the amount was not enough, then all computations made are reverted [6].

7.2 Machine Learning and Blockchain

Both, Blockchain and ML are extremely promising ﬁelds and the combination of them could be even more powerful. Some possible combinations will be discussed in this section.

7.2.1 Smart Contracts using Machine Learning

Smart contracts enable decentralised apps (dApps) which run on a Blockchain. The most famous dApp is ”The DAO” [7]. DAO stands for decentralised autonomous organisation and technically can be anything that is not owned by a central authority. ”The DAO” was created by Slock.it [12] and deployed on the Ethereum Blockchain in 2016. It started with a period of funding in which users could buy tokens, which granted ownership and voting rights, by spending Ether to the contract. After this period ”the DAO” started to work. People were able to submit proposals and token holders could use their voting rights to decide how the money inside ”the DAO” was spent. The result was an amount of $150 million dollars trough 11,000 members over 7 Combining Machine Learning with Blockchain 93 a period of a month. Unfortunately it was the victim of a hack [5]. This caused a fork of the Ethereum Blockchain in Ethereum Hard Fork (ETH) and Ethereum classic (ETC) [3]. One can imagine the DAO as an application built out of smart contracts in the middle and all the shareholders giving inputs on the edges. In such an architecture there are several possible places where ML could be implemented. One is the middle part. ML components could try to learn user preferences and suggest submissions matching these with higher probability and/or priority to the appropriate users. Users are equal to Ethereum addresses in this case. Another possibility is to replace the users on the edges with ML components which decide fully autonomously whether to vote for a submission or not. It is also conceivable that these components on the edges get inputs from other components using ML. So there could arise a complex network out of semi-intelligent applications which all run on a Blockchain and emerge to a highly intelligent system. Once deployed to a Blockchain, no one could stop them without destroying or resetting the whole Blockchain on every node. On the other hand there a several risks to mention at this point. As the Blockchain is designed to be transparent, the behaviour, weights, etc. can be seen by everyone and therefore it may be possible to simulate [2] their decision process and generate adversarial proposals. Just as neural networks can be confused by adversarial images that look unchanged to humans [11], the same may be applied to the proposal input space, which is also of high dimensionality. This is only one example of using ML in smart contracts. There are a lot of more but this combination seems like a ﬁeld in which further research looks promising.

7.2.2 Feature Selection on Blockchain

The ERC20 token standard [29] made it easy to generate tokens which are built upon the Ethereum Blockchain. To make it simple, it is a smart contract containing the balances of every token holder. Currently, there are a total of 77092 ERC20 token contracts based on the Ethereum Blockchain1. Each one of them serves a different purpose and may have smart contracts concerning only that specific token. A ML application which wants to learn from one specific token only needs the information concerning this one. As there is a lot of other information (all contracts and transactions not belonging to the observed token) left, which is noise, an effective way to concentrate on the relevant data and blend out the rest would be helpful. A method to identify the appropriate contracts could lead to automation in the next step. For this purpose, a feature selection method is needed. Variable and feature selection addresses the problem of improving the performance of predictors and can operate on input spaces of several thousand variables. The process tries to find correlating and unnecessary inputs and decrease the amount of information that is needed [10].

1 ETHERSCAN: https://etherscan.io/tokens (last access: 24/04/2018) 94 W. P. von Pilchau

An example could be a token which focuses on electric power distribution from provider to consumer. Considering that the Internet of Things is coming with an increasing number of devices connected to the Internet, the consumer could be a washing machine with the objective to buy the energy as cheap as possible. To archive this goal, it would have a ML algorithm implemented which monitors the appropriate Blockchain and contracts. It would only want to observe those contracts who sell electricity in an area nearby or accessible. From this data, the washing machine tries to learn the time windows when it is cheapest to wash. There exist several startups and companies working on transactive energy and Block- chain technology. Exergy [16] has been the first company who applied Blockchain to a transactive grid by delivering the Broocklyn Microgrid, the world’s first ever energy Blockchain transaction platform. Electron2, a UK startup, applies the Block- chain technology to the energy sector, for building more efficient, resilient and flexible systems. They offer a smart contract-based platform for energy trading, smart meter data privacy and facilities for energy and gas switching [17]. These projects could be used as targets to implement the example. An automation for feature selection on a Blockchain, in particular the Ethereum Blockchain, could be adapted to many ML algorithms and different use cases. Fur- ther research in this field seems promising.

7.2.3 Artiﬁcial Life on a Blockchain

Artificial Life describes life, which is made by humans rather than by Nature. In contrast to biological life, which can only be observed, artificial life can be created. It attempts to recreate biological phenomena from scratch within computers and other artificial media. The intention is to generate life-like behaviour to get a better understanding of how evolution works [15]. Prominent examples of artificial life are Tierra and Avida. Tierra can be described as a basic form of life, comparable to single-celled organism. The system was built in a simulated computer. With the use of genomes which encoded a set of instructions, a generation of artificial organisms was created. One of the instructions was a copy command, which was used to reproduce itself. There was a probability, that, while reproducing, a mutation occurred and the set of instructions was altered. Using this method, Thomas S. Ray, the inventor of Tierra, created a complex evolution with several species and different social behaviour. There was a restricted resource in the environment for which the species had to compete. In Tierra this resource was storage space. Avida is an enhancement of Tierra in which every digital organism has it’s own storage space and is executed by it’s own CPU. Organisms are not able to read or write storage space others than their own. While in Tierra the organisms shared one single brain, in Avida every lifeform has it’s own. Also the virtual CPUs

2 ELECTRON: http://www.electron.org.uk (last access: 24/04/2018) 7 Combining Machine Learning with Blockchain 95 of different organisms can run at different speeds. The speed of a CPU is determined by several factors, but most importantly, by the tasks the organism performs. [4]. It might be promising to create artificial life on a Blockchain. Because of the decentralised character it might be an environment in which artificial life can discover new possibilities and unknown potential. One enhancement of Tierra was the extension of the system to the Internet which represented a new environment with new possibilities for the lifeforms to discover, the same applies to the Blockchain. There is already a kind of artificial life existing on the Blockchain. It’s called plantoids and is described as the plant equivalent of an android. It’s a low level lifeform with one part in the real world and the other part as code in the Blockchain.The real world component is an artwork made of recycled steel and electronics. The virtual element is a smart contract. The creators define the goal of the project on their website3 as follows: to illustrate one of the most revolutionary - and yet still unexplored - aspects of Blockchain technology. It illustrates the ability to create ”Blockchain-based lifeforms”, i.e. algorithmic entities that are: 1. autonomous 2. self-sustainable, and 3. capable of reproducing themselves, through a combination of Blockchain-based code and human interactions. The idea is, that with people funding the plantoid/smart contract, it will gather enough money/coins to first pay an actor who creates the physical representation and then reproduce itself. Applying mutation and genetic algorithms will create new and maybe better plantoids. Another lifeform already living on the Blockchain are the so called ”crypto kitties”4. They can not reproduce themselves and need users to breed new ones, but they are very successful in convincing people to invest money and reproduce them, as one kitty named ”genesis” was sold for 246 Ether, that at time of sale was equivalent to $117,712 dollars [1]. This is important, because as money is the resource needed to execute code on the chain and for that to reproduce, it shows that it was possible for an artificial lifeform to gather enough to theoretical start a whole lot of reproductions. An interesting area of further research may be how reinforcement learning agents behave on the Blockchain. They could be pretrained on the Blockchain history up to the current moment and then try to survive, reproducing themselves using genetic algorithms to improve their code/behaviour. They could try to buy and sell coins to make benefits and use them to reproduce themselves. The earned amount could be used as reward signal. It seems promising to further inspect this topic and create more complex lifeforms or whole generations of them on a Blockchain.

3 PLANTOID: http://www.okhaos.com/plantoids (last access: 23/04/2018) 4 CRYPTOKITTIES: https://www.cryptokitties.co/ (last access: 27/09/2018) 96 W. P. von Pilchau

7.2.4 Creating an Oracle for Smart Contracts using Machine Learning

Smart contracts, by their nature, are able to run algorithmic calculations and store and retrieve data. Because every node runs every calculation, it’s not practical (and presently impossible) to make arbitrary network requests from a contract, which means from inside a Blockchain. Because every node, when validating a block, has to get to the same result of a query, they must not use fluctuating values from the web. It is not guaranteed and also impossible, that a node validates a block at the same millisecond that it was mined, in the elapsed time the result of the query from the Internet could have changed and so the validation would fail. Because of this reason it is only allowed to use in-chain values for smart contracts. Oracles fill this void by watching the Blockchain for events and respond to them by publishing the results of a query back to the contract. In this way, contracts can interact with the off-chain world [18]. Technically, an oracle is a smart contract which is fed from an external service with the suitable data. With the data in the chain, every other smart contract can query these without consistency issues. There is the need to trust the oracle. One of the key advantages of the Blockchain technology is decentralised and using a third party application destroys this feature. But that is only one problem which occurs when using an oracle. There is the subjec- tivity of truth leading to the question which source can be trusted. Some approaches use a median computed out of several different sources delivering the same information. Others deliver one specific source. In the foreseeable future the fact that smart contracts are bound to their in-chain world will not change and so this field seems promising to research on. ML could help to improve the safety and/or the decentralisation of oracles. One could imagine a machine learning algorithm, which develops decision strategies for selecting the best and trustworthiest data source. On the other hand it has to be ensured, that the data the algorithm learns from is secure and trustworthy. Another approach could be an algorithm which learns an aggregation function to apply on the data of several sources. A simple example may be a weighting of the different sources.

7.3 Conclusion

This article gives a broad view of some possible combinations of the Blockchain technology with ML and focuses on the high level concepts. As this research area is new at the time this article is created, there is a lot of potential to explore the discussed topics deeper and in more detail. Also, a lot of research is done by others, so that there can be expected a lot of new achievements in the near future, which will open completely new perspectives and approaches to this area. Finally, there can be said that each of the discussed subjects has its own potential, but more and deeper research is needed in all of them. 7 Combining Machine Learning with Blockchain 97

Next steps include a broader research of related work which tries to combine ML and Blockchain in the one or another way.

References

1. 1 million spent in crypto cats on the ethereum blockchain. https://usethebitcoin.com/1- million-spent-crypto-cats-ethereum-blockchain/, accessed: 2018-09-27 2. Reversing evm bytecode with radare2. https://blog.positive.com/reversing-evm- bytecode-with-radare2-ab77247e5e53?gi=66c4605148f6, accessed: 2018-09-27 3. What is ethereum classic? ethereum vs ethereum classic. https://blockgeeks.com/guides/ what-is-ethereum-classic/, accessed: 2018-09-27 4. Adami, C.: Introduction to artificial life, vol. 1. Springer Science & Business Media (1998) 5. Bharadwaj, K.: Blockchain 2.0: Smart contracts (2016) 6. Buterin, V., et al.: Ethereum white paper. GitHub repository (2013) 7. DuPont, Q.: Experiments in algorithmic governance: A history and ethnography of the dao, a failed decentralized autonomous organization. Bitcoin and Beyond: Cryptocur- rencies, Blockchains and Global Governance. Routledge (2017) 8. Gervais, A., Karame, G.O., Wust,¨ K., Glykantzis, V., Ritzdorf, H., Capkun, S.: On the security and performance of proof of work blockchains. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. pp. 3–16. ACM (2016) 9. Grira, N., Crucianu, M., Boujemaa, N.: Unsupervised and semi-supervised clustering: a brief survey. A review of machine learning techniques for processing multimedia content, Report of the MUSCLE European Network of Excellence (FP6) pp. 1001–1030 (2004) 10. Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of machine learning research 3(Mar), 1157–1182 (2003) 11. Huang, S.H., Papernot, N., Goodfellow, I.J., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies. CoRR abs/1702.02284 (2017), http://arxiv.org/abs/1702.02284 12. Jentzsch, C.: Decentralized autonomous organization to automate governance. White paper (2006) 13. Kotsiantis, S.B., Zaharakis, I., Pintelas, P.: Supervised machine learning: A review of classification techniques. Emerging artificial intelligence applications in computer engineering 160, 3–24 (2007) 14. Lamport, L., Shostak, R., Pease, M.: The byzantine generals problem. ACM Trans. Pro- gram. Lang. Syst. 4(3), 382–401 (Jul 1982), http://doi.acm.org/10.1145/357172.357176 15. Langton, C.G., et al.: Artificial life (1989) 16. LO3 energy: Exergy - technical white paper (2017), https://exergy.energy/wp-content/ uploads/2017/12/Exergy-Whitepaper-v8.pdf 17. Lombardi, F., Aniello, L., Angelis, S.D., Margheri, A., Sassone, V.: A blockchain-based infrastructure for reliable and cost-effective iot-aided smart grids (January 2018), https: //eprints.soton.ac.uk/417098/ 18. Michaelis, R.: Evaluierung und implementierung von orakeln zur bereitstellung von ex- ternen daten fur¨ smart contracts in ethereum 19. Mitchell, T.M., et al.: Machine learning. 1997. Burr Ridge, IL: McGraw Hill 45(37), 870–877 (1997) 98 W. P. von Pilchau

20. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Technical Systems for Survival in the Real World. Springer International Publishing (2018), https://doi.org/10.1007/ 978-3-319-68477-2\ 7 21. Nakamoto, S.: Bitcoin open source implementation of p2p currency. P2P Foundation 18 (2009) 22. Omohundro, S.: Cryptocurrencies, smart contracts, and artiﬁcial intelligence. AI Matters 1(2), 19–21 (Dec 2014), http://doi.acm.org/10.1145/2685328.2685334 23. Peck, M.E.: Blockchains: How they work and why they’ll change the world. IEEE Spec- trum 54(10), 26–35 (2017) 24. Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018) 25. Swan, M.: Blockchain: Blueprint for a new economy. ” O’Reilly Media, Inc.” (2015) 26. Szabo, N.: Formalizing and securing relationships on public networks. First Monday 2(9) (1997), http://ojphi.org/ojs/index.php/fm/article/view/548 27. Tomforde, S., Brameshuber, A., Hahner,¨ J., Muller-Schloer,¨ C.: Restricted On-line Learning in Real-world Systems. In: Proc. of the IEEE Congress on Evolutionary Com- putation (CEC11), held 05 Jun - 08 Jun 2011 in New Orleans, USA. pp. 1628 – 1635. IEEE (2011) 28. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 29. Vogelsteller, F., Buterin, V.: Erc-20 token standard. sept. 2017. URl: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-20-tokenstandard. md 8

Evolutionary Learning for Processing Pipelines in Image Segmentation and Monitoring Systems

Andreas Margraf

Online Process Monitoring Group, Fraunhofer IGCV, Augsburg, Germany [email protected]

Abstract. Recent advancements in machine learning for segmentation and classification has lead to an increase of data processed by various industries. While modern computers manage to handle big data with ease, the effort for preparation and labeling is an open problem. Espe- cially classifier models with several independent variables require relatively large data sets for training. In certain domains though, they are difficult to collect due to limited availability, privacy issues or security constraints. Furthermore, expert knowledge is necessary to design classifier models. Although some tools simplify data preparation, it remains a time-consuming and complex task. Evolutionary algorithms provide means for the selection of filters and labeling by automating feature extraction to a certain extent. Studies show that Genetic Programming (GP) is able to find combinations of operators that segment regions of interest well. Therefore, this article suggests the use of Cartesian Genetic Programming (CGP) for segmentation tasks. The classification of objects in big data is also discussed. Furthermore, the automated design of filters for digital signal processing is presented along with respective fields of application. In this context, industrial optimisation problems like condition monitoring and predictive analysis are presented as part of the research goals. In summary, joining CGP with classifiers allows for substantially reducing design time of machine learning applications.

Keywords: Organic Computing, Cartesian Genetic Programming, Online Process Monitoring, Image Segmentation, Automated Algorithm Conﬁguration, Digital Sig- nal Processing

8.1 Introduction

With increased performance capabilities of modern computer systems a wide variety of application scenarios for machine learning (ML) techniques has opened up in recent years. Although several approaches have made considerable progress in artif- ical intelligence, some practical problems remain unsolved to this date. The rising popularity of embedded devices, e.g., yields large amounts of data. The spread of such information carriers and thereby large sources of data serves online learning in 100 A. Margraf an organic computing (OC) context [28]. OC is considered a design paradigm that comprises nature-inspired algorithms and concepts [20]. Its goal to engineer innovative systems and architectures provides a solid base for multi-agent collaboration and identifies concepts for self-adapting, self-optimising and self-configuring units [27]. Thereby, OC deals with the increasing complexity of information in potentially unstable and unpredictable environments. In practice, a functioning ML approach is supposed to work under time, size or security constraints. Hence, some OC concepts operate with artificial neural networks (ANN) or evolutionary algorithms (EA) which are explained later in this section.

Furthermore, several industries have used intelligent systems to automate large portions of their workflows. Nevertheless, the potential for more efficiency, productivity and cost savings has not been exploited, yet. Quality assurance and process monitoring represent key factors in production management. Thus, many concepts that outsource tasks to humans where automation is not available are already in place for certain domains. Tools that fall under the umbrella of collective intelligence exist, e.g. wikis, crowdsourcing or collaboration systems. However, having data processed solely by human workers implies the risk of social bias, duplicate participation or metholodical weaknesses as discussed in respective publications, cf. [8, 22]. On- line Process Monitoring (OPM) targets highly automated activities that are prone to minor changes by internal and external factors. The complexity of today’s production systems demands appropriate monitoring and fault diagnosis. As efficiency of production systems depends on failure-free operation, monitoring activities must interfere with mechanical elements as little as possible. The main goal of OPM is, thus, to evaluate stability, quality and security of a process [34]. Different types of sensors deliver all kinds of data, e. g. pressure, temperature, speed, sound or images. Acquired information is then analysed, imposing specific expectations to software engineering, architectures and algorithms.

ANNs gained reputation due to their ability to learn tasks such as analysing or classifying big data better than competing techniques without being specifically programmed. They are modelled after the biologoical processes in the brain which raised the idea for the perceptron, cortex and stacked layers. However, performance and accuracy depend on the net topology, design parameters and configuration, all of which have to be defined in advance. Evolutionary Algorithms, in turn, are nature-inspired, flexible heuristics applicable to optimisation, modelling and simulation problems. This class of algorithms borrows concepts from genetics and biology like selection, reproduction, mutation and recombination. A fitness function evaluates the quality of a population. The process, repeatedly applied, mutates parts of the population until a termination criterion is met. In contrary to problem-tailored algorithms, EAs do not necessarily return the global optimum, but likely escape minima in complex search spaces [6]. 8 Evolutionary Learning for Processing Pipelines 101

The remainder of this article is structured as follows: at ﬁrst, an overview on related work is given (Section 8.2) before continuing with outlining the use of an adapted GP and CGP algorithm for data preparation in large data sets (Section 8.3). The relation to competing machine learning techniques is also presented. Subsequently, application scenarios are discussed (Section 8.4) before concluding with a summary and imposing research questions for future work in Section 8.5.

8.2 Related Work

Although neural networks make it possible to master even complex classification problems, certain tasks still remain difficult to automate: the collection and preparation of data. ANNs in particular only reach for high levels of accuracy if trained with large and mainly balanced data sets. Besides, data scientists can utilise software like Labelbox1, LabelID2 or LIBLABEL3 which provide support for image annotation. Amazon Mechanical Turk [3] completely relies on human input. In spite of several available classification algorithms, collecting training data often lacks of automation. In the light of insufficient software support to data labeling, Stein et al. [26] examined the properties of OC architectures combined with a priori knowledge. Since performance and accuracy of neural nets depend on their design, tools exist for deep neural architecture search such as Google AutoML [23]. A refined concept for evolutionary learning called Cartesian Genetic Programming (CGP) was brought up by Miller et al. [18] in the 1990s. In the early days, CGP was successfully applied to the design of circuits for image processing as demonstrated by Sekanina et al. and Harding et al. (cf. [24, 10, 9]). Montes et al. [19] continued the work using CGP for image filtering. In the same context, evolving image features with adapted CGP was examined by Kowaliw et al. [14]. Apart from the development of classification systems, industries tested a number of approaches to establish automated quality monitoring in and around production lines. In this regard, online inspection systems gained importance to ensure safety standards in the aircraft and spacecraft industry, especially focusing on lightweight structures. Here, vision systems are applied to isolate defects of different shape, size, contour and orientation on carbon fiber reinforced plastics (CFRP) as shown by Geinitz et al. [7]. Similarly, Wedel et al. presented a line scan sensor camera system for scan range extension by filtering color channels [32]. The application of OC paradigms to monitoring systems was proposed by Margraf et al. (cf. [16, 15]).

1 Labelbox: https://www.labelbox.com (last access: 12/10/2018) 2 LabelID: https://sweppner.github.io/labeld (last access: 12/10/2018) 3 LIBLABEL: http://www.cvlibs.net/software/liblabel (last access: 12/10/2018) 102 A. Margraf 8.3 Approach

Efﬁcient engineering of monitoring systems requires more self-conﬁguring and self- optimising traits. To pursue this goal, I will suggest adapted evolutionary learning techniques for data segmentation tasks in the context of digital signal processing (DSP).

8.3.1 Preliminary Considerations

The design of algorithms for machine learning requires a model e.g., a probability distribution, that suits the input data well. Usually, a data set has to be divided into partitions for training, validation and testing. For the partition ratio, there are two competing considerations: the variances of parameter estimation and the performance. Both are mutually exclusive. A test dataset usually follows the same probability distribution as the training dataset. To minimise the effect of overfitting, a model should not only fit the training set well but also conform to test data. As for validation, hyperparameters of a model e.g., weights of neurons and number of hidden layers, are tuned. For classifiers such as Logistic Regression (LR), Support Vector Machines (SVM) or ANNs, a deterministic approach with stochastic gradient descent (SGD) is commonly applied. An alternative approach, as proposed in this paper, utilises less problem tailored algorithms. Besides, variance decreases with increasing size of both training and validation set and vice versa. This dependency, of course, affects classifier performance. Unfortunately, large amounts of data raise the effort for collection, preparation and labeling which, in some cases, could make the procedure overly expensive.

8.3.2 Selection of Machine Learning Algorithms

The complexity of real world problems on the one hand and the variety of available ML algorithms call for an intelligent system suggesting a suitable approach. Applications range from anomaly detection, pattern recognition to the classification of objects. Monitoring systems for fault diagnosis and quality assurance deal with recurrent defect detection or classification tasks. Therefore, object geometries or signal patterns have to be segmented and clustered. Associations between a class to promising machine learning techniques can be generated using an abstract yet exact description. This relation is discussed in the paper by Stein et al. [26] dubbed Life Long Learning (L3). In this context, a model-to-algorithm relation is stored in a database. Some insights are borrowed from the field of Automated Algorithm Con- figuration (AAC). In the following section, I examine the potential of evolutionary algorithms for solving special AAC problems. 8 Evolutionary Learning for Processing Pipelines 103

8.3.3 Evolutionary Learning

As one of the most popular nature-inspired algorithms, genetic programming has been used to encode representations of individuals from very different origin. In general, genetic programming was invented to create computer programmes. In a research context, we aim to investigate a heuristic to generate and optimise sensor networks. Modelling those networks with minimal user input requires a probability based initialisation method. The optimisation is then a question of mutation which must be defined beforehand. The evaluation component must be selected wisely to trigger an evolutionary process that reaches high fitness levels. Genetic programming meets all the aforementioned requirements well and therefore has proven useful in sensor network generation. Cartesian Genetic Programming (CGP) denotes a heuristic derived from GP with limitations to inputs, outpus, connections between nodes and number of nodes. It is usually represented by a string of integers. A cartesian programme P is defined as: P := {G,ni,no,nn,F,n f ,nr,nc,l}. The parameters of P contain G as the genotype, whereas f ∈ F represents the function of a node. Every function f also belongs to = ∑n | | one operator class opi, therefore n f i=1 opi . nr and nc denote the number of rows and colums, l stands for the levels-back parameter. The number of inputs is given by ni, the number of connections a node can take by nn, while no defines the number of outputs. The configuration according to the example in Fig. 8.1a is formally defined as follows:

ni = 1,no = 1,nn = 1,n f = 54,nr = 6,nc = 9,l = 1; The deﬁnition of the set of operator classes Op is denoted as follows:

Op := {op1,...,opn}

F also contains the neutral operator ope that has no effect on the underlying values, thereby F := { fi | i = 1...n}∪{ope}. The deﬁnition of the set of hyperparameters p ∈ P is as follows: (1) (m) m P := {pi =(p ,...,p ) | i = 1,...,|Op|∧pi ∈ R }

In general, CGP can evolve solutions efﬁciently for complex problems. Furthermore, it can be modelled to return two or more outputs. The encoding integers represent the function of nodes in the directed acyclic graph (DAG). Genetic drift [30], [17] occurs under certain circumstances, while genetic bloat [29] is avoided due to the constraints to rows and columns. Only nodes connected to a path leading from input to output take effect while nodes not mapping an input to an output node are ignored. These nodes are denoted silent or non-coding. At the same time, CGP exhibits a many-to-one genotype-phenotype mapping. The concept is depicted in 8.1a. Gray circles contain a function f ∈ F accepting one input (nn = 1). The colored arrows indicate possible data ﬂows through the DAG. As for measures to reach higher levels of automation, I propose an evolutionary learning technique for processing 104 A. Margraf

(a)

(b)

Fig. 8.1: Data processing depicted as genotype and phenotype in Fig. 8.1b of a cartesian programme and evolution of the fitness in the best 5 runs (8.1b) pipelines and classification based on previous research [16]. In general, experts tune the parameters of predefined processing pipelines. In the context of varying external conditions, such as sensor focus, camera angles or controller configuration of the assembly line, any change to the parameters causes disruptions in the value distribution which requires heavy modifications. In order to relieve users, the authors of the aforementioned study suggest to automatically tune filter parameters of an image processing programme by analysing the labelled input of a small set of images. As for this approach, the generation of configuration for the filter operators is performed by CGP as presented in this section. 8 Evolutionary Learning for Processing Pipelines 105

(a) (b) (c) (d)

Fig. 8.2: Non-woven fabric and prepreg depicted with labels (8.2a, 8.2c) and segmentation results (8.2b, 8.2d)

The evolutionary progress also strongly depends on the selection of the fitness measure. In previous studies, two fitness scores proved superior to others: Matthew’s Correlation Coefficient (MCC) and F-score. F-score returns a value between 1 (best) and 0 (worst) and is defined as: (1 + β2) · TP Fβ = (1 + β2) · TP+ β2 · FN+ TP MCC returns a value between −1 (worst) and +1 (best) and is defined as: TP· TN− FP· FN MCC = (FP+ FP) · (FP+ FN) · (TN+ FP) · (TN+ FN) While both measures take into account precision and recall, F-score represents the harmonic average of both. MCC also aims to weigh out the binary classification problem even if the classes are of substantially different sizes. Insofar, MCC provides more information on the statistical dependencies between all confusion matrix categories which is beneficial for tuning evolutionary algorithms. Once the basic configuration of this approach was implemented, resulting filters were applied to test images. Figure 8.2 illustrates how the algorithm performed on our data: based on an input image, with a corresponding label, the segmented ROI was computed. The curve of the best five runs generated by the programme are shown in Fig 8.1b. The best fitness returned a value of 0.75. Although this may indicate room for improvement, the returned parameter sets worked well on test sets as can be seen in the resulting images, cf. Figures 8.2a to 8.2d. All in all, the process pipelines applied on images with CGP optimised parameters generate promising results.

8.3.4 Digital Filter Design by Means of Cartesian Genetic Programming

In relation to digital images any digital signal represented by a continuous sequence of real valued numbers can be processed and segmented. Therefore, we aim to apply CGP on digital signals by adapting the definition of input data and the algorithm 106 A. Margraf hyperparameters. For DSP, electronic filters are commonly defined as circuits which manipulate signals depending on their frequency and amplitude in order to remove or attenuate certain parts. Linear filters can be described best by their frequency response. In general, the design of digital filters remains a complex task with several, sometimes contradictory, requirements to be taken into account. There are two types of filters: finite impulse response (FIR) and infinite impulse response (IIR). The following list gives an overview of the restrictions considered in the design process: • the specific frequency response and impulse response • the specific phase shift or group delay • linearity and time-invariance • stability • locality • low computational complexity • particular hardware or software Designing a digital filter can be considered a multidimensional optimisation problem in a complex search space. The selection of filters is similar to the orchestration of processing pipelines as discussed in Section 8.3.2. Of course there are methods in use that help manage these requirements already such a the ParksMcClellan algorithm and the Chebyshev approximation. In the following, we will consider a differing concept for designing a digital filter regarding limitations and particular specifications. As for evolutionary computation, the type of representation has to be defined at first. When looking at audio signals as a frequency, we can easily plot it as a linear dependency between frequency ω and time t. Based on these considerations, evolutionary algorithms are an appropriate concept for automatically arrange filters and further parameters. In this context, previous publications have suggested the use of GA for optimising parameters of an FIR [21, 4] or IIR [33].

8.3.5 Algorithm Conﬁguration and Classiﬁer Modelling

Contrary to CGPIP [24], evolutionary processing pipeline design has not been examined, yet. Therefore, I suggest to apply evolutionary pipeline design to larger sets of filters, thus creating an extended search space. The generation of operator sequences with subsequent parameter tuning is set for future research. For a wide range of applications, I suggest to augment the evolutionary learning approach for the automated design of processing pipelines (ADPP) as presented by Stein et al. [26]. One major challenge relies on the selection of default configurations for hyperparameters and the sequence of operators. Usually, the initial population is generated randomly. However, the evolutionary process might not be able to escape local minima, if the initial configuration performs too badly. In consequence, a solution to 8 Evolutionary Learning for Processing Pipelines 107 select a good first try has to be identified. In image processing, there are standard hyperparameters for regular image filters that have proven useful for rapid development which I will consider for implementation in the short term. Nonetheless, I will pursue the goal to implement a long term learning approach for better initial configurations in future work. For further experiments, the input of the algorithm should be limited to n data objects and corresponding ROI. In the application scenario, segmentation of defects, i.e. isolating surface anomalies, should be provided. An extended set of image filters will be available, so that even difficult segmentation tasks are completed. In this scenario, of course, users are still required to provide data objects, manually draw the ROI and add a label for the classes of e.g., defects. In order to proceed with automation, I suggest to simplify this interaction by segmenting and preselecting the training images or data and require the user to make only slight changes to the ROI.

As for classiﬁcation, the use of neural networks is considered one option for large image datasets. Unfortunately, their performance decreases if training set size remains (too) small. As mentioned before, the proposed approach in this paper aims to build classiﬁers on scarce datasets. However, this concept is not limited to generating image processing pipelines. CGP can be used for creating a neural network topology and tune the weights of nodes as well as transfer functions using an evolutionary learning approach. A certain kind of neural networks even manage to perform a pixel-based segmentation and make predictions for independent ROIs. Thus, image processing pipelines can extend neural network abilites by augmenting training data sets even with few data. As for the design of neural nets, several techniques have been successfully applied. Neural network topology requires a number of hidden layers and weights for single nodes. The common algorithm for this task is stochastic gradient decent (SGD), a greedy optimisation heuristic. CGP, denoted CGPANN, has proven useful as a valid algorithm to generate the topology, node weights and connections for neural networks as demonstrated by Khan et al. [13]. This approach is also known as neu- roevolution. The potential of CGPANN has been demonstrated by applying it on various data sets ([11, 2, 12, 2, 31]). Although my future research will not entirely focus on CGPANN, I will consider it a complementing technique.

8.4 Fields of Application

The automated design of algorithms as well as automated algorithm conﬁguration (AAD, AAC) can be applied in a number of scenarios to prepare heuristics for real world data. In education, simulations illustrate the evolution of populations or genetic constitutions over time in a biological, medical or chemical contexts. Fur- thermore, any task that is usually solved by selecting an algorithm to compute a problem-tailored solution could be automated by unfolding the beneﬁts of ADPP. 108 A. Margraf

A number of industries may benefit in the following application scenarios: labeling for digital image processing, neural network tuning, hyperparameter optimisation and digital filter design. Also, whenever private data cannot be collected easily, CGP can optimise data segemntation. Furthermore, shop floor and job shop optimisation problems, where e. g. machine cells are organised around production lines are solvable through CGP. Finally, creating and configuring sensor networks, architecture and communication patterns is a perfect search space for evolutionary and non-deterministic algorithms. Furthermore, the algorithms can be tested for reliability on publicly available data sets such as the famous Boston Housing or Wisconsin Breast Cancer dataset. For research purposes, the imagenet database [5] with over 14 million images in more than 21000 categories [1] will serve as a data source for the evaluation of the algorithms’ accuracy.

8.4.1 Evolutionary Learning for Signal Processing

Several methods have been proposed to design digital filters e.g., the frequency sampling and window method for FIR filters, pole/zero placement or bilinear z-transform for IIR filters. Those concepts are wide-spread and popular among industrial applications [25]. The use of signal filters which are optimised using a GA - and CGP in particular - is manifold. In general, the design of digital filters needs to solve the trade-off between low filter response, minimum number of filter operations, computational resources and power or time contraints. Especially real-time applications, of course, demand respecting time windows. Finding the optimal setting for a specific application can very well be optimised with an evolutionary approach. The number of operations would be the best candidate for a CGP programme. Further applications include any digital signal emission based on vibration such as: vibration analysis for machine fault detection e.g., bearings in rotating elements of motors. Also, diagnostic systems based on acoustic emissions need an optimised digital filtering. Also, non-destructive evaluation (NDE) heavily makes use of localisation by filtering echoing signals such as ultra sound. At the same time, the technique is used to map surfaces which can be used for quality inspection.

8.4.2 Self-adaption for Digital Filter Design

In the previous sections, CGP was introduced as a general approach to segment data of any kind. For practical use with often unexpected data representations, the concept should adapt to varying conditions. Therefore, future work will focus on the self- adaptive aspect of CGP. The following areas demand ﬂexibility in particular: Machine Condition Monitoring is a means of quality inspection used predomi- nantly in industrial environments to identify critical states. This can provide information on the machine health, meaning inspection and repair slots as well as routine 8 Evolutionary Learning for Processing Pipelines 109

Fig. 8.3: Common signal and sensor types in industrial environments that form a network maintenance can be scheduled more efficiently. In comparison to static maintenance schedules, the use of sensor data for minimisation of efforts and downtime will help save costs. A multi-dimensional signal input can be beneficial for an improved prediction of maintenance slots and anomaly detection. Therefore, a network of sensors is usually set up for a complete monitoring of several parts in the production line. Figure 8.3 illustrates a fictional production line equipped with different types of sensors and their emitting signals that can be processed by a monitoring system.

Noise ﬁltering: Apart from that, vibration and clearance of linked elements are often the origin of heavy noise in the workplace, production facilities, construction sites or even public areas. Since this causes health risks for both employees, passerbies and residents, there is a high priority for sound absorption and suppression. One way to deal with the issue would be installing sound absorbers in or around devices, which is obvious and an established policy. Alternatively, affected individuals could carry sound absorbing ear protectors. The development of noise cancelling strategies is based on digital ﬁlter design, at which point we will come back to evolutionary algorithms to develop them. At the same time, if not sound suppression is the major goal, one can use the frequency information to evaluate the health state of machine elements, such as bearings or similiar elements. In a well functioning device, the sound emissions follow a regular pattern. Any deviation from the usual can be a sign of upcoming problems or malfunctions. This phenomenon should then be utilised to indicate the need for maintenance and repair before the situation becomes critical. In consequence, workers and operators gain time for rescheduling and planning.

Electrical and medical signals: A widespread kind of digital signal in medical applications is the Electroencephalography (EEG), a method to record electrical activity in the brain. Analysing the EEG amplitude lets doctors monitor a patient’s wakefulness. A trained eye may easily evaluate and conclude whether the patient is in an OK state or intervention is necessary. However, the continuous monitoring of any - sometimes vital - signal is tiring and humans tend to perform worse over a 110 A. Margraf long time. Automatic monitoring performed by computers will be more reliable and exact. To accomplish that task, an algorithm would have to ﬁlter the regular pattern in the EEG repeatedly and check for accordance with the health state. Designing that ﬁlter will help saving time.

8.5 Conclusion and Research Questions

Evolutionary algorithms are known for their ability to discover solutions even for complex problems which cannot be solved within polynomial time. Given an optimisation problem, they are useful for parameter search, for learning sequences of filter cascades or complex classification tasks. When searching for solutions to data segmentation, CGP finds the sequence of operators within classes of operators and also approximates hyperparameters by striving for the optimum. Although solutions may not always be deterministically solved, they are still valid and usually applicable to the scenario at hand. In the research work presented in this article, application of CGP to image and signal processing tasks have been examined, focusing on feature extraction in particular. The studies returned promising results. However, the comparability of results is still not sufficient, therefore a general score or measurement unit has to be determined. In addition, the performance of the envisioned algorithm for big datasets with a large variety of operators, classes and their combinations will be studied. In this context, essential parameters of CGP have not been studied enough, namely the type of selection strategy and mutation method. Although previous work suggests a good default setting for CGP, there is still room for improvement, so that scientific proof based on experiments is still missing. In addition, statistical analysis such as significance and hypothesis testing on data sets will be performed in order to scientifically evaluate the potential and limits of the presented approaches. Although the feasibility of evolutionary algorithms in signal processing has been shown in various studies, the proof of concept is still pending. Future work, thus, will cover the automated evolution of database-specific parameters for data object features. Also, the selection of dimension-reducing statistical moments for image processing will be examined. For better precision, an appropriate local pixel neighbourhood size should be computationally defined. Furthermore, the selection of an appropriate number of digital filters for images and digital signals respectively must be identified to solve complex problems. The overall goal is to make ML applications less dependent on expert input which I will especially focus on in my PhD thesis.

References

1. Seeking a better way to ﬁnd web images (Nov 2012), https://www.nytimes.com/2012/ 11/20/science/for-web-images-creating-new-technology-to-seek-and-ﬁnd.html, (last access: 10/02/2019) 8 Evolutionary Learning for Processing Pipelines 111

2. Ahmad, A.M., Khan, G.M.: Bio-signal processing using cartesian genetic programming evolved artificial neural network (cgpann). In: Frontiers of Information Technology (FIT), 2012 10th International Conference on. pp. 261–268. IEEE (2012) 3. Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6(1), 3–5 (2011) 4. Castillo, O., Montiel, O., Sepulveda, R., Melin, P.: Application of a breeder genetic algorithm for system identification in an adaptive finite impulse response filter. In: Evolvable Hardware, 2001. Proceedings. The Third NASA/DoD Workshop on. pp. 146–153. IEEE (2001) 5. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. pp. 248–255. IEEE (2009) 6. Eiben, A.E., Smith, J.E., et al.: Introduction to evolutionary computing, vol. 53. Springer (2003) 7. Geinitz, S., Margraf, A., Wedel, A., Witthus, S., Drechsler, K.: Detection of filament misalignment in carbon fiber production using a stereovision line scan camera system. In: 19th World Conference on Non-Destructive Testing (2016) 8. Goodman, J.K., Cryder, C.E., Cheema, A.: Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making 26(3), 213–224 (2013) 9. Harding, S., Banzhaf, W.: Genetic programming on gpus for image processing. Interna- tional Journal of High Performance Systems Architecture 1(4), 231–240 (2008) 10. Harding, S., Leitner, J., Schmidhuber, J.: Cartesian genetic programming for image processing. In: Genetic programming theory and practice X, pp. 31–44. Springer (2013) 11. Iliadis, L., Papadopoulos, H., Jayne, C.: Engineering Applications of Neural Networks: 14th International Conference, EANN 2013, Halkidiki, Greece, September 2013, Pro- ceedings. No. Teil 1 in Communications in Computer and Information Science, Springer Berlin Heidelberg (2013), https://books.google.de/books?id=Swq7BQAAQBAJ 12. Khan, G.M., Arshad, R., Khan, N.M.: Efficient prediction of dynamic tariff in smart grid using cgp evolved artificial neural networks. In: Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on. pp. 493–498. IEEE (2017) 13. Khan, M.M., Khan, G.M.: A novel neuroevolutionary algorithm: Cartesian genetic programming evolved artificial neural network (cgpann). In: Proceedings of the 8th Inter- national Conference on Frontiers of Information Technology. pp. 48:1–48:4. FIT ’10, ACM, New York, NY, USA (2010), http://doi.acm.org/10.1145/1943628.1943676 14. Kowaliw, T., Banzhaf, W., Kharma, N., Harding, S.: Evolving novel image features using genetic programming-based image transforms. In: Evolutionary Computation, 2009. CEC’09. IEEE Congress on. pp. 2502–2507. IEEE (2009) 15. Margraf A., Geinitz, S., Wedel A., and Engstler, L.: Detection of surface defects on carbon fiber rovings using line sensors and image processing algorithms. In: Proceedings of the SAMPE Europe Conference 2017 Stuttgart (2017) 16. Margraf A., Stein, A., Engstler L., Geinitz, S. and Hahner,¨ J.: An Evolutionary Learning Approach to Self-configuring Image Pipelines in the Context of Carbon Fiber Fault De- tection. In: Proceedings of the 16th IEEE International Conference for Machine Learn- ing and Applications (2017) 17. Miller, J.F., Smith, S.L.: Redundancy and computational efficiency in cartesian genetic programming. IEEE Transactions on Evolutionary Computation 10(2), 167–174 (2006) 18. Miller, J.F., Thomson, P.: Cartesian genetic programming. In: European Conference on Genetic Programming. pp. 121–132. Springer (2000) 112 A. Margraf

19. Montes, H.A., Wyatt, J.L.: Cartesian genetic programming for image processing tasks. In: Neural Networks and Computational Intelligence. pp. 185–190. Citeseer (2003) 20. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing: Technical Systems for Survival in the Real World. Autonomic Systems, Birkhauser,¨ Cham, 1st edn. (2017) 21. Oner, M.: A genetic algorithm for optimisation of linear phase fir filter coefficients. In: Signals, Systems & Computers, 1998. Conference Record of the Thirty-Second Asilo- mar Conference on. vol. 2, pp. 1397–1400. IEEE (1998) 22. Paolacci, G., Chandler, J.: Inside the turk: Understanding mechanical turk as a partici- pant pool. Current Directions in Psychological Science 23(3), 184–188 (2014) 23. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Le, Q., Kurakin, A.: Large- scale evolution of image classifiers. arXiv preprint arXiv:1703.01041 (2017) 24. Sekanina, L., Harding, S.L., Banzhaf, W., Kowaliw, T.: Image processing and cgp. In: Cartesian Genetic Programming, pp. 181–215. Springer (2011) 25. Sekanina, L., Vasˇ´ıcek,ˇ Z.: On the practical limits of the evolutionary digital filter design at the gate level. In: Workshops on Applications of Evolutionary Computation. pp. 344– 355. Springer (2006) 26. Stein A., Margraf A., Geinitz, S., Wedel A., and Engstler, L.: Toward an Organic Com- puting Approach to Automated Design of Processing Pipelines. In: SAOS Braunschweig (2018) 27. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 28. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 29. Turner, A.J., Miller, J.F.: Cartesian genetic programming: Why no bloat? In: European Conference on Genetic Programming. pp. 222–233. Springer (2014) 30. Turner, A.J., Miller, J.F.: Neutral genetic drift: an investigation using cartesian genetic programming. Genetic Programming and Evolvable Machines 16(4), 531–558 (2015) 31. Vanneschi, L., Bush, W., Giacobini, M.: Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics: 11th European Conference, EvoBIO 2013, Vienna, Austria, April 3-5, 2013, Proceedings. Lecture Notes in Computer Science, Springer Berlin Heidelberg (2013), https://books.google.de/books?id=0Va5BQAAQBAJ 32. Wedel, A., Geinitz, S., Margraf, A., Drechsler, K.: Online detection and categorisation of defects along carbon fiber production using a high resolution, high width line scan vision system. In: ECCM17 (2016) 33. Xue, L., Rongchun, Z., Qing, W.: Optimizing the design of iir filter via genetic algorithm. In: Neural Networks and Signal Processing, 2003. Proceedings of the 2003 Inter- national Conference on. vol. 1, pp. 476–479. IEEE (2003) 34. Yin, S., Ding, S.X., Xie, X., Luo, H.: A review on basic data-driven approaches for industrial process monitoring. IEEE Transactions on Industrial Electronics 61(11), 6418– 6428 (2014) Part III

Nature and Social Systems as Inspiration for Organic Computing

Interactive Agent-Based Biological Cell Simulations for Morphogenesis

Andreas Knote and Sebastian von Mammen

Human-Computer Interaction, University of Wurzburg,¨ Wurzburg,¨ Germany {andreas.knote,sebastian.von.mammen}@uni-wuerzburg.de

Abstract. Inspired by the success of interactive simulations in the analysis of complex systems, we investigate interactive simulations of biological systems, especially multicellular organisms. Such interactive biological simulations aid biologists in understanding the behaviour of simulation models, in the search for model inconsistencies, in guiding model development, and in the design and preparation of wet lab experiments. We currently focus our research on the formation of structure and shape of cellular organisms, a process called morphogenesis. Cellular biological systems can be modelled as self-organising systems of cells acting as autonomous agents. In terms of sensors and actuators, chemical messaging and physical interactions between the cells are an essential requirement for modelling the agents’ behaviours. Simulations combining these aspects have high computational demands, and existing simulations in the biological domain often use optimisations that limit their real-time capability which is required for human-computer interaction. In this chapter, we ﬁrst report the key results of an interactive, immersive biological cell simulation prototype. Based on this, we lay out and prioritise the next steps to tackle the problems of complexity, interaction, visualisation, and integration with the workﬂow.

Keywords: Interactive Simulation, Biological Cell Simulation, Complex Systems, Develop- mental Biology.

9.1 Introduction

Empirical sciences strive for accurate and comprehensive models of reality. In biology, such models have inherently great complexity and are hard to retrace. On the one hand, there is a large number of factors whose impact remains to be investigated, and on the other hand, empirical assays are time and resource intensive. Interactive simulations have proven their usefulness in the analysis, development and reﬁnement of complex system models [4, 18]. We aim at transferring this usefulness to biological systems, especially multicellular organisms, supporting biologists in three different ways: (1) Model exploration, (2) model analysis, and (3) synthesis. 116 A. Knote, S. von Mammen

Of the various steps towards meaningful interactive visualisations [11, 21], we have identiﬁed the following four to require special attention in the context of interactive simulations. First, a modelling paradigm for the biological system needs to be selected. Second, the necessary mathematical models that power the simulation and their implementation raise issues of computational demand and scalability, which need to be addressed. Third, meaningful visualisations and interactions need to be designed. Lastly, existing data sources and sinks need to be integrated.

We begin by sketching three ways in which interactive biological simulations can beneﬁt biologists in Section 9.2. The requirements for such interactive systems will be detailed in Section 9.3. In Section 9.4, we introduce related work on biological cell simulations and agent-based simulations. In Section 9.5, we give an overview of a prototypical interactive, immersive biological cell simulation used to obtain feedback from domain experts. We discuss feedback and technical issues and derive the resulting next steps in Section 9.6, before we conclude this work in Section 9.7.

9.2 Interactive Simulations for Complex System Analysis

First of all, interactive simulations allow users to better understand the behaviour of simulation models through interactive exploration of simulation setups, thereby improving the users’ proficiency when applying the model in future experiments. For example, users might be interested in exploring the impact of directed proliferation and over-proliferation on the resulting shape of tissue. Using the proposed interactive simulations, a suitable scenario comprised of the involved entities, such as cells and supporting structure, could be set up quickly. A multitude of possible simulation outcomes can then evaluated by the use, as he or she triggers the events of interest at will and either discards or pursuits the unfolding events. Also, merely observing interactions between different tissue layers, or exploring diffusion gradients of different gene products, or tracing which cells see which signals would allow deepening insights into the dependencies within the complex spatial model of cellular systems and to learn about the transitions of its states. Second, interactive simulations can be used in the analysis of model inconsistencies based on a researcher’s knowledge and experience, thereby guiding model development. As the exploration of the simulation landscape and its dynamics becomes more accessible, the user can effortlessly refocus on possibly unexpected developments during the runtime of the simulation. This process can be supported, for instance, by providing real-world data to measure the outcome of the simulated processes and have real-time visualisations of potential divergences. Ultimately, alleviating the burden of exploring models and continuous refinement of digital models can lead to better accuracy and wider scopes of predictions and 9 Agent-Based Biological Cell Simulations 117 thus to faster convergence of hypotheses and models. This simplifies the synthesis, given sufficiently plausible and tested models, and costly wet-lab experiments can be planned and tested in dry simulations, improving efficiency.

9.3 Requirements of Interactive Biological Simulations

Merks and Glazier [17] make a case for a mesoscale simulation of tissue – treating the simulated tissue as a composite of smaller units, but still abstracting from individual molecules – and promote cells as the appropriate smallest simulated units. To cope with the computational complexity of this approach, the cell model needs to be sufficiently abstract. Treating cells phenomenologically as agents that react to the influences from the environment with a basic set of responses can already be sufficient to explore the dynamics of tissue formation. Chemical messaging and physical interactions between the cells are a crucial requirement for modelling the mechanisms involved in the self-organisation of cells. Cells can produce and sense chemical or electrical signals, which persist and propagate to their neighbours, for example through diffusion processes, which allows them to communicate indirectly by manipulation of the environment (“stigmergy”, [23]). The complexity of biological systems results in computationally very demanding simulations. It is not unusual for biological simulations to require high-performance clusters to run. Consequently, many existing simulations use optimisations that limit their usability in interactive setups or are very hard or impossible to adapt to support dynamic user interactions. Even though millions of cells might be necessary to represent significant portions of organic tissue, meaningful effects in the simulated biological populations can already be observed when only considering several hun- dred or thousands of cells [9, 20]. Still, even at this number, the demands of an interactive simulation are high, and the implementation and hardware requirements rise accordingly as well. Therefore, sleek algorithmic designs, exploiting parallelisation, distributed computing, and efficient algorithms as well as selecting apt optimisations and modelling approaches are paramount. Considerable efforts have to be invested in interfaces that make the user aware of the relevant data in a meaningful way and provide precise and powerful interaction modalities. The demands in terms of visualisation and interaction vary strongly between different simulation models and goals. For example, for the analysis of pressure and strain created by internal forces of a cellular organism, these might be visualised through vector fields or colored overlays, while analysing specific disruptive events might best be identified using break conditions and highlighting of cell states. Lastly, to provide a useful tool for biologists and to support them in their research, it is necessary to integrate the interactive simulation with the existing methodological ecosystem. For instance, the import and use of empirically obtained data, such as volumetric CT data, should be as effortless as possible. 118 A. Knote, S. von Mammen 9.4 Related Work

We provide a short overview of work foundational to our research. As we aim for a niche between existing cell simulations and real-time interactivity, we reference work from several distinct directions. Merks [17] points to the importance of cell-centered models, emphasising the view of biological systems as complex systems. Biological cells and the concept of autonomous agents both draw from high autonomy and low central control. MecaCell [9] provides a framework for the integration of physical and behavioural models into a biological simulation. CellSys [13] and CompuCell3D [22] offer examples of an integrated design and visualisation approach. MecaGen [8] offers a full development environment including behavioural programming through gene regulatory networks. Large-scale agent-based simulations on parallel and distributed platforms are an active area of research [14, 3]. Timothy [6] is a recent example of a large-scale agent-based modelling framework for the simulation of biological cell colonies on massively parallel systems. It can provide up to 109 virtual cells (ca. 1cm3 of tissue). However, these simulations are not interactive in the sense of giving the user the ability to interfere with the simulation state at runtime. An interactive simulation based on the Cellular Potts Model, a lattice-based simulation where a combined energy function controls cell interactions, growth, and movement stochastically, is found e.g. in Morpheus [20]. In the domain of molecular simulations, several works on interactive simulations of the processes of interest have been published. Lv et al. [16] research an interactive molecular visualisation system based on the Unity game engine. Frey et al. [10] developed a generic system to link molecular simulations with interactive front-ends called MDdriver. Molecule position and energy data is sent to the visualisation engine using a network connection, and user input is, in turn, converted into a simulated physical force to be applied to molecules in the simulation. In conclusion, a large body of work already covers the simulation biological systems, and agent-based simulation further gains traction due to its ﬂexibility. Yet, technical limitations have so far limited the development of real-time interactive biological simulations at useful scales.

9.5 Real-Time Interaction Prototype

We have created an interactive simulation prototype that combines a virtual biological cell model with an immersive visualisation [15]. It allows to observe and interact with virtual biological cell agents [12, 17] in a mock-up experiment using virtual reality gear, as shown in Figure 9.1. Based on this early prototype, ﬁrst feedback was obtained from domain specialists. 9 Agent-Based Biological Cell Simulations 119

Fig. 9.1: The interactive simulation prototype as seen through a head-mounted display (cropped ﬁeld of view). (a) Two types of virtual cells are initially placed in an unstable equilibrium, controlled by two competing signaling substances (blue/green) which are produced by the cells. (b) The balance can be easily disturbed by the user by introducing new cells or changing the local substance density, causing a phase of uncontrolled proliferation of one cell type and the extinction of the other. (c) The shape and composition of the cell cluster can be visually inspected throughout the simulation.

9.5.1 Cell Model

Multiple cell types can be deﬁned, each with custom logic programmed in C++ or using a visual scripting language. Cells can interface with their environment by (1) producing substances and executing logical routines based on the measured concentration of substances in their surroundings, (2) controlling their growth, (3) attaching and detaching regions of their surface to/from other cells, (4) active motion, (5) proliferation/division, or (6) cell death (apoptosis). These mechanisms form the basis of bottom-up modelling of the behaviour of multi-cellular organisms with the goal of recreating their emergent properties. The governing parameters might be subjected to manual, automatic, or guided self-optimisation, see for example [7].

9.5.2 Methodology

A GPU-driven physics engine, Nvidia Flex [19], originally developed for visual effects, was repurposed to provide baseline performance. We further implemented a simple diffusion-reaction simulation on the GPU that allows for basic diffusion simulation of multiple substances, based on a grid lattice. The system was not tailored to physical correctness for this use case, but was meant to provide plausible behaviour for the evaluation by domain experts. More speciﬁcally, it had to provide cells that can proactively deform and be deformed by external forces as well. Most importantly, it should allow the user to actively, intuitively and dynamically engage with the simulation. For this prototype, no integration of empirical data was implemented. All interaction takes place using tracked motion controllers. The user can create and remove cells of different types, exert physical forces, and modify the local concentration of chemicals. The density can be visualised using a relocatable 2D plane. An 120 A. Knote, S. von Mammen

Fig. 9.2: Exemplary user interaction sequence: (a) 0s: The initial configuration of two populations of A- and B-cells (green, blue), homogeneously grouped, is mechanically suspended in a rigid frame. Each cell type produces specific signalling substances SA (green) and SB (blue). (b) 2s: The user places a new A-cell near the line of segregation. (c) 3s: The A-cells are proliferating due to the presence of SB. (d). . . (g) 5s...12s: A-cells continue to proliferate and produce SB that causes B-cells to die. (h) 3s: The concentration of SA diminishes, and the proliferation comes to a halt. Interactive simulation prototype, as seen through head-mounted display (cropped field of view). adjustable cutting plane can be used to inspect the interior of a cell cluster. The simulation is capable of simulating up to 250 cells for a virtual reality (VR) simulation (Test Setup: Intel i7-7600K, 16 GiB RAM, Nvidia GeForce GTX1080 8 GiB) with VR visualisation, and maintain 30 frames per second for up to 1024 cells without.

9.5.3 Mock-Up Experiment

We prepared a mock-up experiment containing two types of cells, A- and B-cells. A-cells produce a substance SA that induces cell death in B-cells. B-cells produce a substance SB that induces proliferation in A-cells. Initially, a matrix of homogeneously distributed cells of each type is mechanically suspended in a rigid frame, as shown in Figure 9.1. This conﬁguration quickly settles in an unstable equilibrium, with a gap dividing the two populations. As shown in Figure 9.2, the user can now place new cells or increase the local concentration of SA or SB, to excite the cells and destroy the equilibrium, usually leading to domination of the A-cells in this example. 9 Agent-Based Biological Cell Simulations 121 9.6 Discussion and Next Steps

The prototype of our interactive simulation allowed us to receive feedback from experts in mathematical modelling of developmental systems and experts in empirical developmental biology on multiple occasions. However, we did not perform a formal evaluation, which we deemed too costly and restrictive at such an early stage. We therefore focused on observation of the interactions with the system and anecdotal feedback gathered in subsequent conversations to inform our next steps. Even though the prototype did not aim for scientiﬁc accuracy, the feedback on the ability to “program” cells using visual scripting, and consequently interact with a simulation at runtime was met with interest and support. Exerting forces by pushing cells and triggering disruptive events by distributing signaling molecules in strategic places, thereby guiding development, as well as the ability to look and reach below or behind virtual objects proved attractive to the testers. To obtain the prototype, we tried to re-use existing components from real-time interactive simulations and game engines (e.g. GPGPU-based physics, integration of virtual reality hardware, agent- based entity design, visual programming tools, user interface frameworks). On the one hand, this allowed creating an immersive visualisation with little overhead. On the other hand, it brought a tight coupling between the simulation and visualisation loop, which limits scalability. The use of off-the-shelf physics engines needs to be further scrutinised to assert the validity (in terms of simulation accuracy) and advantages (in terms of adaptability, extensibility and maintainability) of their use. Besides system design and implementation, evaluation and validation of the user interaction design are of high importance. We expect the following objectives to be most important in driving forward the development of real-time interactive biological cell simulations. In order of priority:

(1) Dynamic Coupling of Simulation, Interaction and Visualisation:

Alongside with a selection of other models including ones based on the MecaCell framework [9], we plan to investigate approaches to dynamically coupled simulation, interaction and visualisation. Providing an interface layer that allows communicating the meaningful state data in a compressed manner between simulation and visualisation, and providing a back channel for user interactions, would allow to scale and distribute the simulation setups. Such a loose coupling would allow the experimenter to direct compute time into certain aspects of the simulation (scale or accuracy) and interact/explore/analyse at pre-deﬁned points of interest. The use of (remote) high- performance computing might also alleviate some constraints imposed by current standard workstation hardware. Eventually, offering to switch between different simulation models and/or execution platforms within a single interface, depending on the type of experiment to be conducted, could further extend the range of possible use cases. 122 A. Knote, S. von Mammen

(2) Parameters, Variables and (Visual) Highlighting:

The parameters and state variables of the simulation that are actually of interest to biologists – out of the multitude of parameters that are required to setup the simulation – need to be selectable by the user. Research needs to be done to identify the visualisation methods and metaphors most apt to effectively communicate the state, and highlight relevant state changes, of the simulation. Similar types of variables might then be grouped and categorised to form the basis of a ﬂexible UI framework that might offer to dynamically process and present arbitrary simulation data in a meaningful way. To this end, staying in close contact and establishing fast turn-around times for testing novel prototypes with biologists and related researchers is essential.

(3) (Off-The-Shelf) GPU-Based Real-Time Physics:

The Flex soft body physics simulation, selected for its anticipated speed and flexibility, will need additional scrutiny. The mechanical cell model should be transferred into a stand-alone simulation setup. The introduction of flexible boundaries representing enclosing layers of tissue might allow research in the generation of pressure through cell proliferation and the resulting deformation of tissue. Also, taking lessons from the highly-optimised implementation might benefit the development of strongly GPU-driven, more accurate and interactive physics simulations that could drive more scientifically accurate, future simulations. Both might further benefit the scientific community in using off-the-shelf components for research.

9.7 Conclusion

In this work, we have summarised our motivation for the development of real-time interactive biological cell simulations to support biologists in exploring the complex behaviour of cellular organisms. Seeing cells as autonomous agents and modelling them accordingly provides the path for interactive, high-impact, in-silico exploration of biological systems. We pointed out critical issues along the way to creating such systems. First, a suitable cell model needs to be found that captures all relevant aspects. Agent-based simulations with divergent agent models are computationally expensive and often limit real-time applicability. Nonetheless, such a system should ideally be able to cope with several thousand cells of multiple types or even greater scales. The user interfaces need to be suitable for the investigation by biologists, providing apt interaction methods and visualisations while also scaling with high numbers of cells. Providing a visualisation at the appropriate scale, measurement tools of the appropriate granularity and, especially, tools for the retracing of development over time, are expected to provide high value. The system might offer to replicate different stages of the same organism, or different conﬁgurations of the same protocol, side-by-side. To be useful for biologists, the system needs to integrate with 9 Agent-Based Biological Cell Simulations 123 existing sources of empirical data, such as CT scans. We summarised the results from a ﬁrst horizontal real-time-interactive prototype, which led us to the following three next steps: (1) dynamically couple simulation and interaction/visualisation to improve performance and scalability, (2) elicit the critical parameters and visualisation methods and metaphors, and (3) further investigate of the use of GPU-based computations for interactive biological simulations.

References

1. Proceedings of the European Conference on Artificial Life, vol. 13. MIT Press Journals (2015) 2. Proc. of the Workshop on Software Engineering and Architectures for Realtime Interac- tive Systems (SEARIS). IEEE (2018, in press) 3. Aaby, B.G., Perumalla, K.S., Seal, S.K.: Efficient simulation of agent-based models on multi-gpu and multi-core clusters. Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques (2010) 4. Bell, P.C., O’Keefe, R.M.: Visual interactive simulation — history, recent developments, and major issues. SIMULATION 49(3), 109–116 (1987) 5. Bosman, P.A.N. (ed.): Genetic and Evolutionary Computation Conference, Berlin, Ger- many, July 15-19, 2017, Companion Material Proceedings. ACM (2017) 6. Cytowski, M., Szymanska,´ Z., Uminski,´ P., Andrejczuk, G., Raszkowski, K.: Implemen- tation of an agent-based parallel tissue modelling framework for the intel mic architecture. Scientific Programming 2017, 1–11 (2017) 7. Daschinger,¨ M., Knote, A., von Mammen, S.: An evolutionary approach to behavioural morphometrics. In: Bosman [5], pp. 83–84 8. Delile, J., Herrmann, M., Peyrieras,´ N., Doursat, R.: A cell-based computational model of early embryogenesis coupling mechanical behaviour and gene regulation. Nature Communications 8(13929) (2017) 9. Disset, J., Cussat-Blanc, S., Duthen, Y.: Mecacell: an open-source efficient cellular physics engine. In: Proceedings of the European Conference on Artificial Life [1], pp. 67–67 10. Ferey,´ N., Delalande, O., Grasseau, G., Baaden, M.: From interactive to immersive molecular dynamics. In: Proceedings of the Fifth Workshop on Virtual Reality Inter- actions and Physical Simulations, VRIPHYS, 2008. pp. 89–96 (2008) 11. Fry, B.: Visualizing data: exploring and explaining data with the Processing environment. O’Reilly Media, Inc. (2007) 12. Gelfand, A.: The Biology of Interacting Things: The Intuitive Power of Agent-Based Models: Biomedical applications of ABMs are taking off. Biomedical Computation Re- view 9(3), 20–27 (2013) 13. Hoehme, S., Drasdo, D.: A Cell-Based Simulation Software for Multi-Cellular Systems. Bioinformatics 26(20), 2641–2642 (2010) 14. Kiran, M., Richmond, P., Holcombe, M., Chin, L.S., Worth, D., Greenough, C.: Flame: Simulating large populations of agents on parallel hardware architectures. In: Proceed- ings of the 9th International Conference on Autonomous Agents and Multiagent Sys- tems. AAMAS ’10, vol. 1, pp. 1633–1636. IFAAMAS (2010) 124 A. Knote, S. von Mammen

15. Knote, A., von Mammen, S.: Adaptation and integration of gpu-driven physics for a biology research ris. In: Proc. of the 2018 IEEE 11th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS) [2] 16. Lv, Z., Tek, A., Da Silva, F., Empereur-mot, C., Chavent, M., Baaden, M.: Game on, science - how video game technology may help biologists tackle visualization challenges. PLoS ONE 8(3) (2013) 17. Merks, R.M.H., Glazier, J.A.: A cell-centered approach to developmental biology. Phys- ica A: Statistical Mechanics and its Applications 352(1), 113–130 (2005) 18. Narayanan, S., Kidambi, P.: Interactive simulations: History, features, and trends. Human-in-the-Loop Simulations pp. 1–13 (2011) 19. Nvidia Corporation: Nvidia Flex Physics. https://developer.nvidia.com/ﬂex, last retrieved: 10.1.2018 20. Starruß, J., de Back, W., Brusch, L., Deutsch, A.: Morpheus: A User-Friendly Modeling Environment for Multiscale and Multicellular Systems Biology. Bioinformatics 30(9), 1331–1332 (2013) 21. Steele, J., Iliinsky, N.: Beautiful Visualization: Looking at Data through the Eyes of Experts (Theory in Practice). O’Reilly Media, 1 edn. (2010) 22. Swat, M.H., Thomas, G.L., Belmonte, J.M., Shirinifard, A., Hmeljak, D., Glazier, J.A.: Chapter 13 - Multi-Scale Modeling of Tissues Using CompuCell3D. In: Arkin, A.R.A., P., A. (eds.) Methods In Cell Biology : Computational Methods In Cell Biology, vol. Volume 110, pp. 325–366. Academic Press (2012) 23. Theraulaz, G., Bonabeau, E.: A brief history of stigmergy. Artiﬁcial Life 5(2), 97–116 (1999) 10

Multipotent Systems: A New Paradigm for Multi-Robot Applications

Oliver Kosak

Institute for Software & Systems Engineering, University of Augsburg, Augsburg, Germany [email protected] Home Page: https://www.isse.uni-augsburg.de/

Abstract. Multi-robot applications typically are designed as either homogeneous or heterogeneous systems concerning their software and hardware design. While technologies developed for homogeneous systems provide robustness against failures, they lack versatility concerning applicability. For heterogeneous systems, this observations is true vice versa: While they can be applied to versatile use cases, failures are hard to compensate. We propose the class of multipotent systems to merge the beneﬁts of heterogeneous and homogeneous robot systems. For achieving this, we equip an ensemble consisting of mobile robots with self-organisation abilities enabling it to autonomously adapt itself on coalition level as well as on capability level.

Keywords: Multipotent Systems, Multi-agent, Multi-robot, Self-awareness, Self-adaptation, Self-organisation, Organic Computing, Reconﬁguration

10.1 Motivation

Yet another multi-robot programming framework, are you serious? The last decade has seen a multitude of different applications and frameworks all having the purpose to deploy multi-robot systems for versatile use cases. The spectrum ranges from search and rescue applications [38, 8, 43, 9] over such used for distributed surveillance of critical infrastructure [34, 35, 52] to those dedicated to environmental research [51, 12, 46]. So why is another framework necessary? The sheer number of different applications, robot designs, and programming frameworks shows: For each new use case the wheel is reinvented over and over again, new robots with new capabilities for new applications have to be invented, designed, built, and programmed with individualised software. We see, requirements concerning the capabilities of robots (e.g., measure certain parameters or interact with the environment) typically change in different use cases, but also in the course of operating 126 O. Kosak in a specific one. With the approach proposed in this paper, we aim at overcom- ing this need for specialisation. By separating the concept of capabilities from that of robots in addition to equipping the system with self-adaptation (SA) and self- organisation (SO) abilities, we enable multi-robot systems to be adapted before and autonomously adapt to changing requirements at runtime, following the specification paradigm [20] as known from the Organic Computing domain [47, 37]. We use the terms self-adaptation and self-organisation following the definition of [1]: Self-adaptive systems work in a top-down manner. They evaluate their own global behaviour and change it when the evaluation indicates that they are not accomplishing what they were intended to do, or when better functionality or performance is possible. Such systems typically operate with an explicit internal representation of themselves and their global goals. Self-organising systems work bottom-up. They are composed of a large number of components that interact according to simple and local rules. The global behaviour of the system emerges from these local interactions, and it is difficult to deduce properties of the global system by studying only the local properties of its parts. Such systems do not use internal representations of global properties or goals; they are often inspired by biological or sociological phenomena. To demonstrate the necessity as well as to evaluate the functionality of our framework, we apply our techniques to the case study of ScORe missions [32]. ScORe missions are structured in tasks, where an autonomous system (denominated as ensemble in the following) needs to cooperatively Search for certain parameters, continiously Observe those, and React to critical situations. As running example for such a ScORe mission, the scenario of a chemical accident involving a gas leak is illustrated in Fig. 10.1. Repeatedly, such accidents immensely impair peoples’ daily life (e.g., accident at BASF in Germany, 2016 or at Arkema in Texas, 2017). During the initial Search task, the ensemble has to cooperatively determine the most harmful gas type as well as its source. In the subsequent continuous Observation task, the dissemination of the detected gas has to be tracked while also critical infrastructure (e.g., hospitals) needs to be monitored by the ensemble. If the gas reaches a critical area, in a React task the ensemble has to alert endangered civilians. Like Fig. 10.1 illustrates, in each task other capabilities are required which have to be provided by the robots participating in the task. For fulfilling these requirements, current approaches [43, 9, 34, 12] introduce appropriately configured robots (concerning hardware and software) for each task. This strategy has multiple drawbacks: D1) Resource and engineering overhead: The amount of needed robots grows pro- portionally with the amount of different tasks and use cases. D2) Tradeoff between robustness or versatility: On the one hand, within heterogeneous systems (consisting of robots each with individual hardware and software), the heterogeneity in robots leads to reduced robustness against failures, i.e., if a specialised robot breaks down, it has to be replaced by an equivalently configured one. 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 127

Fig. 10.1: Example of a ScORe mission. The upper part serves as a legend. The lower part shows possible ensemble conﬁgurations and task assignments in the course of that mission (dependencies ordered from left to right, including parallel execution) [25].

Homogeneous systems (consisting of robots equal to each other concerning their hardware and software) on the other hand are specialists in solving one dedicated task and can not be used within others. D3) Planning and allocation complexity: Creating plans that deﬁne the procedure of execution for tasks is NP-complete [13] while allocating those tasks to agents is np-hard worsened through “larger team sizes and greater heterogeneity of robots and tasks” [16]. To exemplify how we want to overcome these drawbacks, we concretise our objectives in Section 10.2. Achieving these objectives, multiple technologies have to be developed and integrated with already existing ones. In this paper, we focus on the software architecture necessary for supporting all of them. In Section 10.3, we illustrate how our concept of a layered architecture supports achieving these objectives and how the needed technologies and algorithms are situated within different layers. Thereby, we constitute how we delimit our approach from those of others. In Sec- tion 10.6, we give an overview of results we found so far as well as an outlook for the next topics we will investigate in the near future.

10.2 Objectives

The objective of our work is threefold as we want to overcome the three main drawbacks of current approaches introduced in Section 10.1. To achieve this we introduce the new system class of multipotent systems, merging the beneﬁts from current homogeneous and heterogeneous systems. 128 O. Kosak

The term multipotency is an analogy from biology which fits very smoothly into the context, as multipotent progenitor cells are able to adapt their functionality when brought into new environments [18]. The new system class of multipotent systems is characterised by its homogeneity concerning the software running on each of the system’s participants (homogeneity at design time) and its adaptable heterogeneity concerning their provided capabilities (heterogeneity at runtime). By this, benefits from both kinds of systems are inherited. On the one hand, with their run time heterogeneity, agents of multipotent systems can be configured very individually to reproduce the versatility of heterogeneous systems [31]. On the other hand, with their design time homogeneity, agents in multipotent systems easily can compensate for each other in case of failures as it can be done in homogeneous systems. These characteristics are exploited to tackle D1–D3: For D1, we exploit the ability to reconfigure the ensemble’s robots in terms of their capabilities. In comparison to related literature on autonomous, homogeneous or heterogeneous multi-robot systems [14, 46, 33, 10], we do not need to provide appropriately configured robots for each of these tasks but enable the system to adapt to changing requirements in reconfiguration phases (cf. R1, R2, R3 in Fig. 10.1). This approach reduces the needed amount of robots. In our example in Fig. 10.1, a conventional heterogeneous system would need 24 devices (10 devices in S,3+5 additional differently configured devices in cO, and further 6 differently configured devices in Re). With our approach exploiting the reconfiguration capabilities of robots, we only need 10 devices that can be configured appropriately in each phase. To tackle D2 by increasing robustness, we propose appropriate SA-SO abilities to compensate for failures of different kinds which cannot be planned for at design time (e.g., break down of robots or sensors). As homogeneous systems are robust against interferences per definition [4], this characteristic can also be proposed for multipotent systems. To achieve an autonomous-as-possible execution of user-defined tasks, we need to further endow our robots with SA-SO abilities which reduces the number of needed interactions between user and ensemble to a minimum. To enable versatility of the system, we propose SA-SO mechanisms such as coalition formation [45] or resource allocation [7] to adapt to unforeseen situations (e.g., changed environmental conditions). To address D3, we reduce the problem of multi-agent planning to that of single-agent planning [28] by considering the ensemble as one entity (the collective acts as a single agent). Thereby, we map the problem of multi-robot task allocation (MRTA) [16] from the class of multi-robot tasks, multi-task robots (MR-MT) to the class of single- robot tasks, multi-task robots (SR-MT) [16]. This is possible in multipotent systems when collective algorithms originally designed for homogeneous systems (e.g., swarm robotic algorithms [4, 49, 42]) implement the actions used for task execution. Actions of individuals do not have to be planned in such algorithms, but only to be triggered in the form of local rules that collectively produce a desired emergent effect. Our objective is to create plans for ScORe missions on ensemble instead of agent level. These tasks then should be executed collectively by using SA-SO 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 129 abilities of the ensemble, where possible (otherwise, we have to fall back on classic planning). This strategy facilitates the act of planing for collectives so that plans are calculated straightforward and adapted at run time [28]. Summed up, our multi-robot programming framework for multipotent systems eases the act of programming, employing, and maintaining ensembles.

10.3 Methodologies

To implement our approach that comes by the drawbacks introduced in Section 10.2, we propose a layered software architecture to be implemented by all robots in the ensemble. This architecture supports all necessary functionalities needed to enable a multipotent system. Each layer encapsulates its own concepts and algorithms designed to enable SA-SO (cf. Fig 10.2). In the following, we ﬁrst introduce our architecture including a speciﬁcation of purposes for each of its layer. Second, we give a brief overview of algorithm classes needed to enable SA-SO. For all algorithms, we assume that robots are able to communicate without restrictions (i.e., through wireless connections such as WiFi or LTE).

10.3.1 Layered Multi-Agent System Architecture

To enable modularity and adaptability needed to come by D1 – D3 with SA and SO (cf. Section 10.2), we model each of our robots as a Jadex Active Components Frame- work Platform [5] and denote it as agent a ∈ A of our ensemble A. Besides offering a communication middleware convenient for many SA and SO functionalities, this framework offers the possibility to dynamically load and unload components from a platform. We exploit this feature to build the core feature of our approach: Dynam- ically exchanging capabilities of robots. From a software perspective, each of the robots in the ensemble is equal to the others (homogeneity at design time). We show the representative software design of one of these platforms (robot a1) in Fig. 10.2. In this layered architecture, we assign special functionalities to each layer, dedicated to solving the user-deﬁned task on a different level of abstraction. In Fig. 10.2, we further depict needed communication in terms of inter-platform communication with other system participants {a1,...,an+1} and communication with the system’s user that introduces the current ScoRe mission. To obtain modularity, communication is restricted to only happen between vertically (intra-/ inter-platform) and horizontally adjacent layers (inter-platform). In the following the different layers, their functionality and interaction is presented in detail. Task Layer: The task layer serves as relay between the system’s user and the ensemble. As the user deﬁnes a new ScORe mission, this mission is broadcasted to the ensemble (no dedicated robot has to be selected by the user). Robots receiving the mission will start to analyse it and try to decompose ScORe tasks for the ensemble. 130 O. Kosak

ScORe mission user design manual

ܽଵ ܽଶ … ܽ௡ instruction

task layer ScORe task synchronization … planning automated automated

ensemble layer ensemble program coordination …

SA-SO agent layer agent program communication agent program… execution

semantic multipotent ensemble hardware layer capability … semantic

plug & play & plug hardware

Fig. 10.2: Layered system architecture focusing on robot a1 of a multipotent ensemble A and its interactions (solid arrows) with the user as well as other agents {a2,...,an}∈A. Dashed arrows symbolise an instantiation on the ensemble, agent, and capability layer, valid for executing a ScORe mission. For a speciﬁc ScORe mission, typically multiple ScORe Tasks have to be solved. Ensemble programmes coordinate multiple agents. Agent programmes may rely on multiple capabilities which themselves may require multiple hardware modules.

Therefore we use Hierarchical Task Networks (HTN) [15] that are especially handy for autonomous systems deployed in real-world environments. In an adapted version we propose, HTN offer a promising combination of external user control (through designed decomposition of complex tasks into primitive tasks) and an appropriate degree of freedom to enable autonomous decisions within the executing ensemble. Consequently, ScORe missions are represented as complex tasks in the HTN while the user-defined task is the root node of the network. After one robot has achieved a valid decomposition, information about the resulting ScORe tasks (i.e., the primitive tasks in the HTN) is distributed within the ensemble. This step is asynchronously done in parallel by all robots and thus multiple (different) decompositions may occur. The ensemble has to synchronise results and determine one solution to be executed (cf. Fig. 10.2). Execution then takes place in the ensemble layer and its subordinate layers. Ensemble Layer: For every ScORe task there is at least one appropriate action available to solve it on ensemble layer, called ensemble programme. An ensemble programme encapsulates the logic for the cooperative execution of the associated ScORe task. We implement ensemble programmes as SA-SO algorithms when possible, e.g., associating swarm algorithms like Particle Swarm Optimisation (PSO) [53] to ScORe tasks of class S. If no appropriate swarm algorithm can be found, we 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 131 fall back on classical multi-agent planning [40] to create valid action sequences for solving the ScORe task. Thereby, we often exploit the advantage SA-SO offers compared to traditional multi-agent planning [28]: Actions of participants do not have to be planned but are deposited as local rules on the agent layer (e.g., despite every robot in the ensemble is working only according to its local rules of the PSO, the collective achieves to find a global optimum as an emergent effect of the cooperation). The ensemble layer thereby is responsible for coordinating the distributed execution. To increase robustness, the existence of multiple ensembles in parallel is possible which work on the same or on a different task simultaneously. Additionally, we allow horizontal inter-platform coordination on ensemble layer, e.g., to exchange agents among ensembles when necessary. To enable ensemble programmes to be suitable for not only one special ScORe task but for a whole class, we design them to be parameterisable. All ScORe tasks of class S, e.g., can be handled by the same algorithm where the parameter of interest can be injected. For instance, the source of gas X in gas accidents like in Fig. 10.1, the highest temperature in case of fire accidents, or highest pollen concentration in an environment monitoring scenario. Agent Layer: The responsibilities on the agent layer are twofold. To execute ensemble programmes, agents in an ensemble have to cooperate with each other. Thus, while coordination of those cooperations is handled on the ensemble layer, the agent layer is responsible for executing the appropriate local rules. Therefore, for all ensemble programmes an accompanying agent programme containing those local rules is implemented on agent layer. As exchanging information between cooperating agents is often necessary within SA-SO algorithms (e.g., exchanging measurements in PSO), we allow horizontal inter-device communication on the agent layer. The second responsibility of the agent layer is to maintain a robot’s self-awareness on its current capabilities and thereby its competence to participate in concretely parametrised SA-SO algorithms (PSO for gas X obviously needs participating robots to provide the capability to measure gas X). Thereby, the agent layer together with the ensemble layer and capability layer is responsible to detect situations that need the agent or even the whole ensemble to be reconfigured concerning their provided capabilities. Capability Layer: The capability layer encapsulates the self-awareness functionalities of each robot. Enriched by a common knowledge base that stores relations between hardware modules and capabilities with robot specific knowledge (e.g., concerning a robot’s geometric design), the capability layer manages the set of available capabilities in any possible robot configuration. This information is essential for the agent layer to determine its competence of participating in a coalition for a specific SA-SO algorithm. Further, this information can be used to propose beneficial reconfigurations of the robots’ hardware in situations where this is needed. Moreover, the capability layer is responsible for executing capabilities, i.e., is the adapter between software representation and hardware execution. When an agent programme on the agent layer triggers the execution of a capability, the capability layer forwards this to the appropriate hardware, collects up possible callbacks (e.g., measurements), and informs the agent programme about the execution status. Thereby, the capabil- 132 O. Kosak 1 user

[ScORe tasks left] [else] plan ScORe mission [else] [success] task layer activate set task status ScORe task [requirements fulfilled] solved select ScORe task

[else] select ensemble allocate program ensemble program [else] [task [else] finished] run ensemble

layer form ensemble program [else] ensemble ensemble [agents left] [agents left] calculate agent

a reconfigurations agent check local rules execute agent update local rules run agent synchronize [else] executability reconfiguration executability program execution state layer agent [caps left]

determine capab. update capability register execution availability executability finished [cap executable]

[cap not exec.] suggest layer reconfiguration

semantic hardware semantic access hardware recognize new execute stop capability description hardware capability execution [stop event]

execute physical 2

user reconfiguration

Fig. 10.3: Example of a ScORe mission execution. Activities on the task and ensemble layer are executed only on a coordinating robot while activities on the agent, the capability, and the hardware layer are executed on all devices. Currently, the physical reconﬁguration has to be accomplished by a user (dashed lines) that also introduces new ScORe missions to start the overall activity. ity layer encapsulates concrete hardware implementations from their usage in higher layers. Semantic Hardware Layer: To enable the capability layer to deduce available capabilities from certain hardware conﬁgurations, as well as to realise actual execution on real hardware, the semantic hardware layer is implemented as an adapter to the hardware. The implementation of that layer is not in the scope of this work, nonetheless, the interface to it is presented in [50].

10.3.2 Algorithms and Technologies

To enable SA-SO abilities of the ensemble and increase its autonomy, we integrate algorithms from three classes that empower the agent to fulfill its responsibilities defined in the different software layers. Basic Ensemble Algorithms: We identified algorithms for forming coalitions (i.e., ensembles) consisting of appropriately configured robots for given tasks with a market-based algorithm in previous work [30] (cf. form ensemble on ensemble layer 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 133 in Fig. 10.3). Moreover, we suggested to resolve conflict situations (e.g., multiple coalitions are found while only one shall activate the task, cf. activate ScORe task on the task layer in Fig. 10.3) by leader election in previous work [30] that can also be achieved in a distributed fashion, e.g., formulated as consensus problem [21], which we are currently investigating in. To enable the self-awareness functionalities of robots (e.g., needed for determine capability availability on capability layer in Fig. 10.3), we design capabilities as agents within the Jadex Active Components Framework [5] combined with semantically annotated hardware modules [50]. To let robots safely cooperate, e.g., within bound areas, we adopt the collision avoidance algorithm of [3] that works distributively and with only local sensors. Planning and Executing ScORe missions: To enable HTN-planning on the task layer, we provide an appropriate domain description language working with actions (ensemble programmes) on the ensemble layer. This enables the design of reusable, parameterisable HTNs which can be composed for different ScORe missions in a modular fashion. To execute ensemble programmes, we similarly classify them according to the ScORe task class they are associated with. Thereby, we develop ensemble programmes implementing appropriate SA-SO algorithms for each class (cf. select ensemble programme on ensemble layer in Fig. 10.3). For S, we identify PSO to be fitting for continuously distributed parameters like gas or temperature. For dis- cretely distributed parameters (e.g., objects) an adapted potential field method [27] can be used where entities distribute equally in a given area to enable the observation of the whole area. The potential field method is also a candidate to be implemented for ScORe tasks of class cO. When, e.g., a cloud of gas X has to be observed, observing robots push each other off as much as possible while additionally staying where a given gas concentration can be measured. Thereby, the tracking of the gas cloud can be seen as an emergent effect. Algorithms for the ScORe class of Re are very specific, as an infinite set of possible situations may happen during S and cO. We, therefore, identify reactions that are commonly needed in most of the situations and find appropriate SA-SO algorithms to implement them, e.g., formation flight based on local rules for guided boiding [41] (control-one-move-all) to evacuate all robots from an area. Robust and Autonomous System: For achieving robustness and autonomy in the ensemble, the ensemble has to be able to compensate for failures of robots as well as failures in hardware providing capabilities to robots. For solving the first, we currently develop an approach to compensate for hard failures (break down of robots) as well as for soft failures (battery depletion) with a transactional task execution system using an extended coalition formation algorithm that allows for exchanging robots among ensembles. For providing the second, we developed a portfolio approach (cf. resource allocation portfolio approach in Fig. 10.4) for solving the multi-agent resource allocation problem (MARA) [7] to reconfigure robots within the ensemble (cf. calculate agent reconfigurations on ensemble layer in Fig. 10.3) that combined with a task allocation algorithm (cf. task allocation and task assignment in Fig. 10.4) maintains the ensembles operability (cf. Fig. 10.4 [25]). Thereby, we are able to determine resource (hardware modules) allocations in the ensemble to fulfill all task 134 O. Kosak

Fig. 10.4: An integrated approach for task and resource allocation (from [25]). Resource allocation takes place when task allocation fails to re-establish a productive system configuration (allocate, assign, and execute tasks). We use a portfolio approach to select an appropriate mechanism (centralised or distributed). requirements concerning needed capabilities that have to be provided by robots (we exploit the possibility to decompose the calculation of a solution to the resource allocation problem like previously done in the energy domain [29]). Consequently, when there is no valid solution to the task allocation problem, the ensemble is reconfigured to allow a subsequently restarted task allocation (cf. task allocation in Fig. 10.4), assign (cf. task assignment in Fig. 10.4) and execute (cf. task execution in Fig. 10.4) the tasks afterward (cf. the situations marked with R in Fig. 10.1, where the systems needs to be reconfigured according to changed task requirements).

10.4 Evaluation

We already achieved some proof of concepts for the practicability of our approach in simulation as well as in real-world experiments. Real World Experiments: In the domain of environmental research, we evaluated our system in its respective state during ScaleX Campaigns 2015 [51] and 2016 [23, 26] and demonstrated its functionalities during the FAS*2016 conference [30, 24]. During ScaleX 2016, an ensemble of up to five mobile robots were autonomously coordinated by our software to achieve a large-scale temperature profile with the Distributed Temperature Sensing (DTS) technology. Thereby, we proofed our concept of controlling robots with Jadex as well as a set of basic ensemble algorithms (self-awareness, coalition formation, leader election, and cooperative execution [30]). For realising the experiment, we instructed the ensemble to cooperatively carry a fiberglass cable along which temperature measurements can be made within every 10 centimeters. The cable was carried along a pre-scripted route in the experiment area in a height of approximately 50 meters autonomously by a heterogeneous ensemble consisting of quadrocopters as well as mobile ground robots. Task allocation, assignment, and the coordinated execution were performed in a SA-SO fashion, 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 135

10m 10m 10m

50m 50m 50m

Fig. 10.5: Proof of concept from the environmental measurement campaign, where a heterogeneous ensemble consisting of four mobile robots achieved an airborne large-scale temperature measurement using a fiberglass cable for Distributed Temperature Sensing measurements in a height of approximately 50 meters. respecting the robots different available capabilities (quadrocopters are able to move to positions higher than 1 meter, mobile ground robots are not) and the tasks required capabilities (some tasks needed to reach up to 50 meters). A sketch of the experiment setup including a proof of concepts achieved in this experiment can be found in Fig. 10.5. Simulated Experiments: We also achieved some first results in simulation concerning the use of suitably applied and parameterisable swarm algorithms for the ScORe task pattern. Figure 10.6 shows the progress of executing the particle swarm optimisation algorithm in our simulation environment that can be applied to S tasks in the course of a ScORe mission. The robots, therefore, need to be able to freely move in 3-dimensional space (“flying”) as well as measuring the parameter of interest. Robots use their self-awareness ability on the capability layer (cf. Section 10.3.1) to determine their qualification for the task. Measurements are propagated in the ensemble and the moving vector is adjusted according to the PSO rules (information on the currently measured maximum concentration of the parameter, the local concentration at a robot’s current position, and a certain random component to avoid local optima). Thereby, robots that cooperate on the agent layer by executing the appropriate PSO-specific agent programme collectively execute the PSO-specific ensemble programme that is coordinated on the ensemble layer. This leads to the correct execution of a S task on task layer, requiring. In addition, we also achieved first results in executing a potential field algorithm suitable for continuous observations and guided boiding suitable for react tasks (cf. Section 10.3.2) in our simulation environment.

10.5 Related Work

Many projects investigate mobile agents and how they can support emergency management (e.g., Search and Rescue (SAR) scenarios) or environmental research, which 136 O. Kosak both can be classified as ScORe missions. In the following, we discuss related approaches and point out their potential as well as their draw-backs in the context of applying them for handling ScORe missions. Current approaches can roughly be classified into two groups: (1) approaches, that try to map principles found in nature to the development of systems consisting of a multitude of simple robot agents (called Swarm Robotics [4]), and (2) approaches, that use traditional AI techniques (e.g., planning, reasoning, learning, scheduling, etc.) [43, 2, 22]. Approaches of group (1) are characterised by the fact that agents only have very limited capabilities and are highly specialised for certain tasks which limits their applicability for ScORe missions, as different ScORe tasks typically require agents with heterogeneous capabilities. Projects following (1) can mainly be found in the domain of environmental/climate research. Here, the usage of swarm robotic algorithms is more common than in SAR scenarios as these systems are often only used to collect data but not to make complex online decisions based on it. For instance, in CoCoRo (2011 - 2014) [44] as well as in its subsequent project subCULTrob (since 2015) [46] autonomous swarms of unmanned underwater vehicles (UUVs) and unmanned surface vehicles (USVs) (all specialised in their capabilities) use swarm robotic principles for ecological monitoring, search, maintenance, exploration, and harvesting in an underwater environment. Other projects focus on developing mechanisms for solving single specialised tasks, e.g., distributed environmental monitoring using mobile USVs. In this domain, the NAMOS project (2003 - 2007) [6] uses coordination done by a multi-agent system, whereas the project CORATAM (since 2014) [12] relies on controllers synthesised with evolutionary algorithms. All approaches classified in type (1) have in common that they reach their goals by using algorithms relying on very simple rules (e.g., finding consensus [48], cooperative transport [11, 36], or clustering [12]), that can only be achieved by the collective as a whole but not by single agents (often described as emergent effect [19]). On the one hand, this emergence brings robustness and fault tolerance, as misbehaviour of single agents can easily be compensated for within a homogeneous swarm where all agents are identical. On the other hand, in complex applications involving multiple tasks (e.g., ScORe missions), problems occur after achieving the goal the swarm robotic system is trained for, as these systems rely on additional guidance if interdependent decisions or actions must be made [4]. In addition, agents used in subsequent tasks usually need to have different capabilities which is often not considered in these approaches, as the swarms usually consist of homogeneous agents. Approaches of group (2) appear to be more appropriate for complex scenarios. The Swarmanoid project (2006 - 2010) [10] combines multiple mechanisms from (1), e.g., to solve a precisely pre-defined SAR mission using heterogeneous agents, but therefore relies on explicitly scripted behaviour. Other projects for handling SAR scenarios often combine intelligent autonomous agents with different capabilities for supporting humans in the SAR process. For example, the SHERPA project (2013 - 2017) [33] focuses on human-robot-interaction with UAV and unmanned ground 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 137

simulated gas source

initial ensemble ensemble during position PSO execution

(a) initial (b) mobile robots used

3 ensemble stabilizes at source 2

1 distance to source gas in meters

0 25 50 75 100 125 150 175 time in seconds (c) measurements, top down view (d) measurements, side view

Fig. 10.6: (A) shows the system previous to execution with the gas source indicated as coordinate system (top left) and all quadcopters in their initial circle formation. In (B) all quadcopters start to execute the local rules of PSO to cooperatively approach the gas source. (C) indicates the state where the gas source is found by the ensemble (no collision avoidance is respected in this simulation run). (D) shows the average distance of the quadcopters to the gas source in meters (during one execution of the PSO) including the standard deviation as error bars, continuously reduced with passed time. vehicles (UGVs). The SWARMIX project (2011 - 2014) [14] combines human capabilities with those of animals (e.g., dogs), as well as artificial agents (e.g., UAV) to create heterogeneous SAR systems. Systems designed for SAR scenarios consisting of artificial agents only are often restricted in their autonomy, as they are only able to react to predefined events. In SINUS (2013 - 2015) [43] or AirShield (2008 - 2011) [8] a wireless connection can be autonomously established over a long distance to stream data from an area of interest to a distant operator for further evaluation. This evaluation and an appropriate reaction have to be determined by the human user of the system. 138 O. Kosak

All of the described approaches have in common that they are heavily specialised for their dedicated application and lack the ability to autonomously adapt to changing requirements (i.e., new capabilities are demanded from agents), which is very important in ScORe missions. While certain approaches use SA and SO to adapt to a changing environment, the possibility of reallocating the agents’ capabilities is neglected. This hinders ongoing approaches to actually adapt to changing ScORe tasks in ScORe missions or switch from one ScORe mission to another (e.g., from environmental monitoring to SAR). Our approach instead allows for both kinds of adaptation. By combining SA and SO principles with planning, we suggest a new paradigm of instructing an ensemble. Being able to reallocate capabilities seems to be the next step to enable real-world applications. We, therefore, will make this additional degree of freedom utilisable and consequently will consider it throughout all software layers. The resulting system architecture will allow robot ensembles to deal with ScORe missions in general.

10.6 Conclusion & Outlook

We presented a layered software architecture integrated with necessary algorithms to enable multipotent systems merging the benefits of both, homogeneous and heterogeneous systems. Therewith, robots implementing the proposed software, homogeneous at design time, can become heterogeneous specialists during runtime. We already realised a basic version of that architecture [32, 28] and integrated necessary algorithms for coalition formation [32], distributed leader election, information distribution [30], coordination of collective behaviours [30], self-adapting and self- organising reconfiguration of robot capabilities [25], and controlling mobile robots (UAVs and ground robots) in general [51, 30]. Our current and next steps include the design and implementation of 1) a domain specific language for specifying modular hierarchical task networks (HTN) in the domain of ScORe missions and 2) of an adaptive, distributed, and transactional algorithm to transfer task execution knowledge in the case of failure for increasing the robustness of the system. To 3) achieve consensus within the ensemble, e.g., during synchronisation of different planning results on the task layer (cf. Section 10.3.1), we will appropriately adapt a leader election approach [17] meeting all needed requirements defined of our system class. We further will automate the act of planning by enabling our system to learn HTNs [39] from data provided by experts so that learned valid plans are suggested to the systems user.

References

1. Babaoglu, O., Shrobe, H., eds.: First IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO 2007) 9-11 July, Boston, MA, USA. Available at http://projects.csail.mit.edu/saso2007/, accessed on 2018-11-15 (Jul 2007) 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 139

2. Becker, M., Blatt, F., Szczerbicka, H.: A Multi-agent Flooding Algorithm for Search and Rescue Operations in Unknown Terrain, pp. 19–28. Springer Berlin (2013) 3. van den Berg, J., Lin, M., Manocha, D.: Reciprocal Velocity Obstacles for real-time multi-agent navigation. In: 2008 IEEE Int. Conf. on Robotics and Automation. pp. 1928– 1935 (May 2008) 4. Brambilla, M., Ferrante, E., Birattari, M., Dorigo, M.: Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence 7(1), 1–41 (2013) 5. Braubach, L., Pokahr, A.: Developing distributed systems with active components and Jadex. Scalable Computing: Practice and Experience 13(2), 100–120 (2012) 6. Caron, D., Stauffer, B., Darjany, L., Oberg, C., Pereira, A., Das, J., Heidarsson, H., Smith, R., Smith, E., Seubert, E., et al.: Networked aquatic microbial observing systems: An overview. Center for Embedded Network Sensing (2009) 7. Chevaleyre, Y., Dunne, P.E., Endriss, U., Lang, J., Lemaitre, M., et al.: Issues in multiagent resource allocation. Informatica 30(1) (2006) 8. Daniel, K., Dusza, B., Lewandowski, A., Wietfelds, C.: AirShield: A system-of-systems MUAV remote sensing architecture for disaster response. In: Proc. 3rd Annual IEEE Systems Conf. (SysCon) (2009) 9. De Cubber, G., Serrano, D., Berns, K., Chintamani, K., Sabino, R., Ourevitch, S., et al.: Search and rescue robots developed by the european icarus project. In: 7th Int. Workshop on Robotics for Risky Environments. Citeseer (2013) 10. Dorigo, M., Floreano, D., Gambardella, L.M., Mondada, F., Nolﬁ, S., Baaboura, T., et al.: Swarmanoid: A Novel Concept for the Study of Heterogeneous Robotic Swarms. IEEE RAM 20(4), 60–71 (2013) 11. Dorigo, M., Tuci, E., Groß, R., Trianni, V., Labella, T.H., Nouyan, S., et al.: The swarm- bots project. In: Int. Workshop on Swarm Robotics. pp. 31–44. Springer (2004) 12. Duarte, M., Costa, V., Gomes, J., Rodrigues, T., Silva, F., Oliveira, S.M., Christensen, A.L.: Evolution of Collective Behaviors for a Real Swarm of Aquatic Surface Robots. PLoS ONE 11(3), 1–25 (03 2016) 13. Erol, K., Hendler, J., Nau, D.S.: HTN planning: Complexity and expressivity. In: AAAI. vol. 94, pp. 1123–1128 (1994) 14. Flushing, E.F., Gambardella, L.M., Caro, G.A.D.: A mathematical programming approach to collaborative missions with heterogeneous teams. In: 2014 IEEE/RSJ Int. Conf. on Intell. Robots and Systems 15. Georgievski, I., Aiello, M.: An Overview of Hierarchical Task Network Planning. CoRR abs/1403.7426 (2014), http://arxiv.org/abs/1403.7426 16. Gerkey, B.P., Mataric,´ M.J.: A formal analysis and taxonomy of task allocation in multi- robot systems. The Int. Journ. of Robotics Res. 23(9), 939–954 (2004) 17. Gharehchopogh, F.S., Arjang, H.: A survey and taxonomy of leader election algorithms in distributed systems. Indian Journ. of Science and Technology 7(6), 815 (2014) 18. Giorgetti, A., Marchetto, M.C., Li, M., Yu, D., Fazzina, R., Mu, Y., et al.: Cord blood- derived neuronal cells by ectopic expression of Sox2 and c-Myc. Proceedings of the National Academy of Sciences 109(31), 12556–12561 (2012) 19. Goldstein, J.: Emergence as a construct: History and issues. Emergence 1(1), 49–72 (1999), http://dx.doi.org/10.1207/s15327000em0101 4 20. Gudemann,¨ M., Nafz, F., Ortmeier, F., Seebach, H., Reif, W.: A speciﬁcation and construction paradigm for organic computing systems. In: 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems. pp. 233–242 (Oct 2008) 21. Hamann, H.: Towards swarm calculus: urn models of collective decisions and universal properties of swarm performance. Swarm Intelligence 7(2), 145–172 (2013) 140 O. Kosak

22. Hussein, A., Adel, M., Bakr, M., Shehata, O.M., Khamis, A.: Multi-robot task allocation for search and rescue missions. Journ. of Physics: Conf. Series 570(5), 052006 (2014), http://stacks.iop.org/1742-6596/570/i=5/a=052006 23. ISSELabs: Flying robot ensemble in action at the ScaleX 2016 geographic measurement campaign. Available at https://youtu.be/MWNyUymtNSs, accessed on 2018-11-15 (Oct 2018), https://youtu.be/MWNyUymtNSs 24. ISSELabs: Saso 2016 - decentralized coordination of heterogeneous ensembles using jadex. Available at https://youtu.be/G8JHShUlQY0, accessed on 2018-11-15 (Oct 2018), https://youtu.be/G8JHShUlQY0 25. J.Hanke, Kosak, O., Schiendorfer, A., Reif, W.: Self-organized Resource Allocation for Reconfigurable Robot Ensembles. In: 2018 IEEE 12th Int. Conf. on Self-Adaptive and Self-Organizing Systems (Sept 2018) 26. KIT IMK/IFU, G.P.: Scalex. Available at https://scalex.imk-ifu.kit.edu/, accessed on 2018-11-15 (Oct 2018), https://scalex.imk-ifu.kit.edu/ 27. Koren, Y., Borenstein, J.: Potential field methods and their inherent limitations for mobile robot navigation. In: Robotics and Automation, 1991. Proceedings., 1991 IEEE Int. Conf. on. pp. 1398–1404. IEEE (1991) 28. Kosak, O.: Facilitating Planning by Using Self-Organization. In: 2017 IEEE 2nd Inter- national Workshops on Foundations and Applications of Self* Systems (FAS*W). pp. 371–374 (Sept 2017) 29. Kosak, O., Anders, G., Siefert, F., Reif, W.: An Approach to Robust Resource Alloca- tion in Large-Scale Systems of Systems. In: Self-Adaptive and Self-Organizing Systems (SASO), 2015 IEEE 9th Int. Conf. on. pp. 1–10 (2015) 30. Kosak, O., Wanninger, C., Angerer, A., Hoffmann, A., Schierl, A., Seebach, H.: Decen- tralized Coordination of Heterogeneous Ensembles Using Jadex. In: 2016 IEEE 1st Int. Workshops on Found. and Applications of Self* Systems (FAS*W). pp. 271–272 (2016) 31. Kosak, O.: A Decentralised Swarm Approach for Mobile Robot-Systems. In: Organic Computing: Doctoral Dissertation Colloquium 2015. vol. 7, p. 53. kassel university press GmbH (2015) 32. Kosak, O., Wanninger, C., Angerer, A., Hoffmann, A., Schiendorfer, A., Seebach, H.: Towards Self-Organizing Swarms of Reconfigurable Self-Aware Robots. In: Found. and Appl. of Self* Sys., IEEE Int. Workshops on. pp. 204–209. IEEE (2016) 33. Marconi, L., Leutenegger, S., Lynen, S., Burri, M., Naldi, R., Melchiorri, C.: Ground and aerial robots as an aid to alpine search and rescue: Initial sherpa outcomes. In: Safety, Security, and Rescue Robotics (SSRR), 2013 IEEE Int. symposium on. pp. 1–2. IEEE (2013) 34. Martinez, C., Sampedro, C., Chauhan, A., Campoy, P.: Towards autonomous detection and tracking of electric towers for aerial power line inspection. In: 2014 Intern. Conf. on Unmanned Aircraft Systems (ICUAS). pp. 284–295 (2014) 35. Matikainen, L., Lehtomaki,¨ M., Ahokas, E., Hyyppa,¨ J., Karjalainen, M., Jaakkola, A., et al.: Remote sensing methods for power line corridor surveys. ISPRS Journ. of Pho- togrammetry and Remote Sensing 119(Supplement C), 10 – 31 (2016) 36. Mondada, F., Gambardella, L.M., Floreano, D., Nolfi, S., Deneuborg, J.L., Dorigo, M.: The cooperation of swarm-bots: physical interactions in collective robotics. IEEE Robotics Automation Magazine 12(2), 21–28 (2005) 37. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 10 Multipotent Systems: A New Paradigm for Multi-Robot Applications 141

38. Murphy, R.R., Tadokoro, S., Nardi, D., Jacoff, A., Fiorini, P., Choset, H., Erkmen, A.M.: Search and Rescue Robotics. In: Siciliano, B., Khatib, O. (eds.) Handbook of Robotics. Springer (2008) 39. Nejati, N., Langley, P., Konik, T.: Learning hierarchical task networks by observation. In: Proceedings of the 23rd Int. Conf. on Machine Learning. pp. 665–672. ICML ’06, ACM, New York, NY, USA (2006) 40. Nissim, R., Brafman, R.I., Domshlak, C.: A general, fully distributed multi-agent planning algorithm. In: Proc. of the 9th Intern. Conf. on Autonomous Agents and Multiagent Systems: Volume 1 - Volume 1. pp. 1323–1330. AAMAS ’10, Intern. Foundation for Autonomous Agents and Multiagent Systems (2010), http://dl.acm.org/citation.cfm?id= 1838206.1838379 41. Reynolds, C.W.: Flocks, herds and schools: A distributed behavioral model. ACM SIG- GRAPH computer graphics 21(4), 25–34 (1987) 42. Rubenstein, M., Cornejo, A., Nagpal, R.: Programmable self-assembly in a thousand- robot swarm. Science 345(6198), 795–799 (2014) 43. Scherer, J., Yahyanejad, S., Hayat, S., Yanmaz, E., Andre, T., Khan, A., et al.: An Au- tonomous Multi-UAV System for Search and Rescue. In: Proc. of the First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use. pp. 33–38. DroNet ’15, ACM, Florence, Italy (2015) 44. Schmickl, T., Thenius, R., Moslinger, C., Timmis, J., Tyrrell, A., Read, M., et al.: Cocoro – the self-aware underwater swarm. In: 2011 Fifth IEEE Conf. on Self-Adaptive and Self-Organizing Systems Workshops. pp. 120–126 (2011) 45. Shehory, O., Kraus, S.: Methods for task allocation via agent coalition formation. Art. Intelligence 101(1), 165 – 200 (1998), http://www.sciencedirect.com/science/article/pii/ S0004370298000459 46. Thenius, R., Moser, D., Kernbach, S., Kuksin, I., Kernbach, O., Elena Kuksina, E., et al.: subCLUTron: a learning, self-regulating, self-sustaining underwater society/culture of robots. Art. Life and Intell. Agents Symposium, 2016 (2016) 47. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 48. Valentini, G., Hamann, H., Dorigo, M.: Efficient decision-making in a self-organizing robot swarm: On the speed versus accuracy trade-off. In: Proc. of the 2015 Int. Conf. on Autonomous Agents and Multiagent Systems. pp. 1305–1314. AAMAS ’15, Int. Found. for Autonomous Agents and Multiagent Systems, Istanbul, Turkey (2015), http://dl.acm. org/citation.cfm?id=2772879.2773319 49. Vas´ arhelyi,´ G., Viragh,´ C., Somorjai, G., Tarcai, N., Szor¨ enyi,´ T., Nepusz, T., Vicsek, T.: Outdoor flocking and formation flight with autonomous aerial robots. In: 2014 IEEE/RSJ Int. Conf. on Intell. Robots and Systems. pp. 3866–3873 (2014) 50. Wanninger, C., Eymller, C., Hoffmann, A., Kosak, O., Reif, W.: Synthesising Capabili- ties for Collective Adaptive Systems from Self-Descriptive Hardware Devices - Bridging the Reality Gap. In: 8th Int. Symp. On Leveraging Appl. of Formal Methods, Verification and Validation (Sept 2018) 51. Wolf, B., Chwala, C., Fersch, B., Garvelmann, J., Junkermann, W., Zeeman, M.J., et al.: The SCALEX Campaign: Scale-Crossing Land Surface and Boundary Layer Processes in the TERENO-preAlpine Observatory. Bulletin of the American Meteor. Soc. 98(6), 1217–1234 (2017) 52. Zhang, J., Liu, L., Wang, B., Chen, X., Wang, Q., Zheng, T.: High Speed Automatic Power Line Detection and Tracking for a UAV-Based Inspection. In: 2012 Intern. Conf. on Industrial Control and Electronics Engineering. pp. 266–269 (2012) 142 O. Kosak

53. Zhang, Y., Wang, S., Ji, G.: A comprehensive survey on particle swarm optimization algorithm and its applications. Mathematical Problems in Engineering 2015 (2015) 11

Decision Support with Hybrid Intelligence

Adrian Calma1 and Dominik Dellermann2

1 Intelligent Embedded Systems Lab, University of Kassel, Kassel, Germany [email protected] 2 vencortex, Kassel, Germany [email protected]

Abstract. Startups are the fresh blood of our economy and, besides universities, are thriv- ing innovation. One of the first steps in this enriching journey to success, where they create jobs and new products, is getting funded. Decision makers, such as investors, incubators, and accelerators, receive a large amount of funding proposals. Currently, new ventures are evaluated “manually”, often without any machine learning or any decision support at all. The implicit rules that every experienced decision maker has developed through years of failures and successes can not be codified, let alone be explained. It is intuition. We envision a learning system that is able to learn the intuition from multiple decision makers and guide them in their judgment. Our envisioned learning system relies on hybrid intelligence, which we define as the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other.

Keywords: Collective Intelligence, Hybrid Intelligence, Collaborative Interactive Learning

11.1 Introduction

Consider the following example: An investor evaluates funding proposals from hun- dreds of startups. Based on an intensive analysis of the submitted documents a decision is made. This is time consuming and biased towards ones believes. Thus, the decision is not 100% objective, as the investors rely on their intuition and “gut feel- ing”. Such decisions are highly complex, as they are made under extreme uncertainty. Despite its limitations (i.e., bound rationality), human intuition is the best that we have to make predictions under extreme uncertainty. For such decisions traditional statistical methods fail, as they can not deal with the dynamic context and the need to annotate soft data such as the creativity of a new venture idea. In case of a fund, there is more than one investor that analyses the funding requests, whereas the set of analysed funding proposals may be disjoint. That is, what investor 144 A. Calma, D. Dellermann

Fig. 11.1: Investors provide data on startup proposals and their judgment as well as proﬁle information on their experience and investment history. The data is analysed, and a personalised forecasting is returned. The investor has also access to the collective intelligence, which is a forecasting based on data from all investors.

A analyses is not necessary analysed by another investor B. Therefore, following questions arise: 1. How can individual decisions be debiased? 2. How can we make use of the heterogeneous knowledge of domain experts? 3. How can we share implicit knowledge without loosing competitive advantages? 4. How can we efficiently and effectively provide decisional guidance for en- trepreneurial decision makers? 5. How can we interpret the decision of our learning model in such a way that it is human readable? We envision a new way to learn from the collective intelligence (or intuition) of various domain experts to support decision makers and to provide high-quality decisional guidance, i.e., hybrid intelligence for artificial intuition. We, therefore, adopt the hybrid intelligence definition from [6] “the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other.” A learning system with hybrid intelligence capabilities offers decision makers 1. personalised guidance based on their previous decisions, which helps them be more consistent in their decisions and 2. guidance based on aggregated knowledge from multiples sources (e.g., investors, domain experts, etc.), which helps them to reduce bias The decision support workflow is depicted in Figure 11.1: All startup data that the three investors received are submitted to the learning system. As we can see, the number of submitted requests is not the same over the three investors. The proportions can differ significantly between individual angel investors or large venture capital companies. Nevertheless, our proposed approach analyses the submitted data and 11 Decision Support with Hybrid Intelligence 145 provides each investor with a personalised insight regarding its current data. This is done based on the history of previous decisions, as the more consistent the decisions are the better the outcome. In other words, we want to have consistent predictions of the future success of the ventures. Moreover, the decision makers can benefit from the collective intelligence. They have access to forecasting values learned on data previously submitted by all system users (i.e., investors, domain experts, historical start-up data). Thus, the decision makers benefit from the experience and intuition of their peers and domain experts. Aggre- gating heterogeneous knowledge is required to make better predictions under such extreme uncertainty [9]. The basic rational of aggregating the judgment of multiple experts is that it reduces potential errors of individuals (e.g., biases or limited cognitive capabilities) [21].

11.2 Related Work – Humans-in-the-Loop in Machine Learning

Although, the terms of AI and machine learning give the impression that humans become to some extent obsolete, the machine learning pipeline still requires lot of human interaction such as for feature engineering, parameter tuning, supervising the training, or developing the learning models. The human is still heavily in the loop for sense-making and training, even though her effort for manual feature engineering has been reduced due to the rise of deep learning [14] and automation approaches (e.g., AutoML [12]), which support human experts in tuning models. For instance, unsupervised learning requires humans to make sense of clusters that are identiﬁed as patterns in data to create knowledge [13]. More obviously, human input is required to train models in supervised machine learning approaches, especially for creating training data, debug models, or train algorithms such as in reinforcement learning [23]. This is especially relevant when divergences of real-life and machine learning problem formulations emerge. This is for instance the case when static (ofﬂine) training data sets are not perfectly representative of realist and dynamic environments [19]. Moreover, human input is crucial when models need to learn from human preferences (e.g., recommender systems) and adapt to users or when security concerns require both control and interpretability of the learning process and the output [10]. Therefore, more recent research has focused on interactive forms of learning (e.g. [17, 1] and machine teaching (e.g. [31]). Those approaches make active use of human input (e.g. active learning [29]) and thus learn from human intelligence. This allows machines to learn tasks that they can not yet achieve alone [18], adapt to environmental dynamics, and deal with unknown situations [2].

11.2.1 Hybrid Intelligence

In [6], we argue that the most likely paradigm for the division of labor between humans and machines in the future is hybrid intelligence. While in contrast with other 146 A. Calma, D. Dellermann publications on decision support and expert systems [17, 16], they emphasise that applications which are able to deal with real-world problems require a continuously collaborating socio-technological ensemble integrating humans and machines, not just using the humans in certain parts and at specific times during the process of creating machine learning models. The Hybrid Intelligence uses the complementary strengths of human intelligence and AI to behave more intelligently than each of the two would do it in separation (e.g., [18]). The basic ideas is exploit the complementary strengths of heterogeneous intelligence (i.e., human and artificial agents) into the socio-technological ensemble. Thus, the hybrid intelligence systems are defined as systems that have the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results than each of them could have done in separation and continuously improve by learning from each other [6]. By performing tasks collectively, the activities conducted by each part are dependent, but not necessarily aligned to reach the same goal (e.g. teaching an AI adversarial [15] tasks such as playing games). The hybrid intelligent system achieves a performance that none of the involved parts could have achieved on its own. Thus, such a system is more efficient and effective on socio-technical level, and, therefore, it achieves superior results compared to its parts. Over time, the hybrid intelligent system improves its and the performance of each of its consisting components (i.e., humans and machines) by learning from each other continuously. Thus, the performance of such systems is measured not only by a superior outcome but also by the learning progress of its parts. In a nutshell, human and AI parts of a hybrid intelligence system, as a socio-technical ensemble, co-evolve to improve over time.

11.3 Problem

We look at the problem from two perspectives: From the perspective of investment decision makers and from the perspective of entrepreneurs seeking for funding.

Investment Decision Makers

The decision makers have to evaluate a large number of funding requests3. Ideally, they have access to a system that performs an automatic screening to provide efficient decisional guidance. The startups are scored based on the funding proposal, team data, news, current market situation, etc. To evaluate startups in this vein, decision makers require highly specialised and heterogeneous knowledge that they might not possess for each specific startup. Also, there is a need for visualising these very heterogeneous startup data. Furthermore, the learning system should provide decision support: On the one hand, personalised decision support for a specific decision

3 For example, UnternehmerTUM [33] receives over 100 (pre-)incubation startup and tech projects per year and has a dealflow of over 1000 startups per year 11 Decision Support with Hybrid Intelligence 147 maker. That is, the decision indication is individually tailored based on the previous decisions, preferences, and risk profile of an investment decision maker. On the other, a second decision indication considers all the available information (i.e., data from all decision makers), thus providing a decision support based on hybrid intelligence. Moreover, the decision provided by the learning system has to be presented to the human decision makers in an interpretable. In this way the prediction is trusted by and justifiable to potential stakeholders.

Entrepreneurs

The startups also beneﬁt from using the learning system. Entrepreneurs get feedback regarding their team constellation, the value proposition, marketing strategy, or their business model. Receiving such feedback provides them with informative guidance which is an indicator for the current status of the venture and allows entrepreneurs to draw conclusions on what to do next. Moreover, in the long run, the learning system based on hybrid intelligence is capable to offer suggestive guidance, i.e., suggesting an entrepreneur a concrete sequence of actions based on the current state of a startup. Decisional guidance for entrepreneurs proved to be highly valuable to increase decision quality, effectiveness, and learning [7].

11.4 Machine Learning Perspective

At first, we focus on the personalised guidance: learning the intuition behind how the decision makers evaluate the most important characteristics that define the concept ”startup”, such as team, value proposition, idea, etc. In the preprocessing phase, we have to reduce the feature space to the most informative features, while keeping in mind that they have to be interpretable for humans. Moreover, we have to deal with heterogeneous input data. For example, in case of scoring the teams on a scale from 1 to 10, we have to statistically summarise different aspects of the team constellation, such as average, coverage, heterogeneity, etc. Another possible solution is to create an ensemble of different models [25]. That is, one model for each team size. The cases mentioned above, we are neither dealing with a typical regression problem nor with a typical classification problem. As the score of a team takes values on a scale from 1 to 10, where the order is significant, we are dealing with an ranking learning problem [30] (also known as ordinal regression [22]). Obviously, we have to quantify the costs, as we have to deal with an imbalanced learning problem (e.g., 70% of tech startups fail [32]). Thus, we have to adapt methods from cost-sensitive learning [11] in order to quantify the possible losses and wins of an investment portfolio. 148 A. Calma, D. Dellermann

Secondly, our learning system provides hybrid collective guidance: integration of multiple knowledge sources. That is, we have apply and adapt labeling fusion techniques [20] for our purpose of deriving collective knowledge from different types of sources, such as multiple investors and domain experts, news, or social media. Certainly, in order to support entrepreneurs in building a successful venture (which might also help in ranking the startups), we have to develop a similarity measure, specially tailored for startups. Startups as input data are hard to reduce to numbers (which we humans can easily understand and compare), whereas no reduction reduces the understanding and a fast comparison. Therefore, we envision the startup blueprint: Each core feature score is allocated a fixed pixel in the blueprint. The score value associated with that specific feature determines the color: The values is transformed to a color value (i.e., 0 to 255). In this way, humans can develop of sense of how a successful startup looks like and assess the different aspects of the core features in a visual way. Nevertheless, we have to continuously improve the performance of our leaning system. Our models have to self adapt to new markets, new development fields, and investor requirements. Thus, we are in a highly dynamic context, where we have to exploit various knowledge [4] and automatically adapt the machine learning models [28]. We envision a future system that is able to effectively and efficiently manage a fund: it evaluates teams, makes requests, makes suggestions for startups, evaluates the progress, and autonomously invests. Obviously, we can not totally exclude humans, but we can reduce their workload, while training the human expert to make better decisions. By providing the human expert with feedback we aim at developing a socio-technical system that can provides better prediction that a single human or machine could offer in separation [8]. By applying active learning techniques [5], we can acquire precise knowledge for the most informative aspects of the current market, new values and trends, and other relevant information.

11.5 Preliminary Results – Predicting the Next Funding Phase

We have conducted a case study to see if we can predict the next funding phase of startup. Our goal was to see how good we can predict which teams are good and which are not. A good team of a startup will reach the next funding phase. We decided to base our prediction only on the team. For this purpose, we analysed 137 153 teams consisting of 1 to 7 members, where each team member of the team was described by 22 features. Based on the team constellation, we trained a binary clas- siﬁer that predicted whether the team was good or not , i.e., the startup described by its team will make it to the next round or not. Together with PhD candidates and postdocs from the Information Systems department at the University of Kassel, 11 Decision Support with Hybrid Intelligence 149 we applied methods of weak supervision and data programming in order to generate the labels for the startups. This was necessary, as we wanted to simulate the case where the label is not known and cannot be determined, thus learning without ground truth [3].

11.5.1 Cost Sensitive Classiﬁer

We used the scikit-learn [24] decision tree implementation to train our classiﬁer. For training, only 20% of data was used, whereas the rest of 80% is used to report the performance of our approach. We implied a randomised search, to tune our parameters, where the maximum depth of the decision tree was varied between 3 and 400, the maximum number of features to consider when looking for the best split between 1 and 16, the minimum number of samples required to split an internal node between 2 and 11, the minimum number of samples required to be at a leaf node between 2 and 11, and the function that measures the quality of a split could either be the Gini impurity or the information gain [26]. The parameters were optimised by a 3-fold cross validation and sampled 50 times. In real life, we might deal with two types of investors: a risky and a careful one. The risky investors do not want to miss a good opportunity and take more bad investments into account. On the other hand, a careful one, would rather miss an investment opportunity then doing a bad investment. In order to simulate these two types, we deﬁned different costs for bad investments and missed opportunities. For the risky investor, we set the cost for misclassifying a bad team as good (i.e., bad investment) to 1 and the cost for missing an opportunity (i.e., misclassifying a good team as bad) to 5. For the simulated careful investor, we switched the costs: a bad investment implies a cost of 5, whereas a missed opportunity a cost of one. Of course, correct predictions did not involve any costs.

Table 11.1: Accuracy for different team sizes. always “bad” and “always “good” are baseline classiﬁers that only predict the “bad” or the “good” class. The last column shows the performance of our classiﬁer for a risky and a careful investor.

our prediction Team Size always “bad” always “good” risky careful 1 97% 3% 99% 99% 2 80% 20% 93% 91% 3 49% 51% 79% 85% 4 42% 58% 76% 72% 5 37% 63% 72% 63% 6 32% 68% 74% 78% 7 27% 73% 62% 69%

The performance of our approach is presented in Table 11.1. For comparison, we presented the results of two other base line classiﬁers: one that always predicts that at 150 A. Calma, D. Dellermann team is “bad” and another one that predicts the opposite, the team is always “good”. As we can notice, the distribution of the two classes is highly imbalanced for teams consisting of 1, 2, 5, 6, and 7 members. Nevertheless, with our approach the success rate can be increased. We notice, that the higher the number of team member, the worse we perform. Even though the distribution of the two classes is similar for team of 5, 6, and 7 members, the performancce decreaases with the increasing number of team members. We theorise, that this has to do with the choice of the classiﬁer and has to be further investigated.

Table 11.2: Accuracy for the most certain 100 teams that were predicted as “good”.

our prediction Team Size risky careful 1 100% 100% 2 77% 95% 3 70% 88% 4 72% 94% 5 79% 90% 6 78% 85% 7 71% 76%

But, nobody has infinite investing capacity. Thus, we decided to evaluate our approach by considering the top 100 teams for the which the classifier is most certain regarding a “good” team. As we can expect, the classfier trained for the careful investor performs better, as the cost matrix models exactly this behaviour. Still, we have a good performance, if take into consideration that 90% of tech startups fail. With our approach, a maximum of 29% of the predictions (and investments) are wrong.

11.6 Conclusion and Outlook

By implying a hybrid intelligence approach, we showed that we can learn even though there is no ground truth. Still, the following question my arise: Did the PhD candidates and Postdocs learn anything? We can report that most of them reported that they learned about their prejudices and by analysing teams, they were enforced to reflect about them and their decision process. Of course, that was not the focus of the study, but we can see that in a hybrid intelligent system, all parts are continuously learning. Furthermore, in order to help the humans (e.g., domain experts) to learn and perform better, we would like to provide feedback regarding the current performance and, presenting them with an explanation of the decisions made by the system. A first step 11 Decision Support with Hybrid Intelligence 151 in that direction is to make the classifier interpretable, e.g., by applying lime [27], a method that tries to explain what machine learning classifiers are doing. Our next step in reaching the envisioned system that is able to effectively and efficiently manage a fund, we would like to specifically select the team (or startup) and the person to evaluate it. By embedding active learning [29] and collaborative interactive learning [5] we can select the next team to be evaluated and, at the same time, the adequate person for the task, thus, answering the question of who should label what?.

References

1. Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) 2. Attenberg, J., Ipeirotis, P., Provost, F.: Beat the Machine. Journal of Data and Informa- tion Quality 6(1), 1–17 (2015) 3. Beyer, C., Bieshaar, M., Calma, A., Heck, H., Kottke, D., Wurtz,¨ R.: Learning without ground truth. In: Organic Computing – Doctoral Dissertation Colloquium 2017. Bochum, Germany (2017), [accepted] 4. Calma, A., Kottke, D., Sick, B., Tomforde, S.: Learning to learn: Dynamic runtime exploitation of various knowledge sources and machine learning paradigms. In: SISSY 2017: 4th International Workshop on Self-Improving System Integration. pp. 109–116. Tucson, AZ (2017), [accepted] 5. Calma, A., Leimeister, J.M., Lukowicz, P., Oeste-Reiß, S., Reitmaier, T., Schmidt, A., Sick, B., Stumme, G., Zweig, K.A.: From active learning to dedicated collaborative interactive learning. In: 29th International Conference on Architecture of Computing Sys- tems. pp. 1–8. Nuremberg, Germany (2016) 6. Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., Ebel, P.: The future of human-ai collaboration: A taxonomy of design knowledge for hybrid intelligence systems. In: Proceedings of the 52nd Hawaii International Conference on System Sciences (2019) 7. Dellermann, D., Lipusch, N., Ebel, P.: Developing design principles for a system for crowd-based business model validation. In: Design Science Research in Information Systems and Technology (DESRIST) (2017) 8. Dellermann, D., Lipusch, N., Ebel, P., Popp, K.M., Leimeister, J.M.: Finding the uni- corn: Predicting early stage startup success through a hybrid intelligence method. In: International Conference on Information Systems (ICIS) (2017) 9. Dellermann, D., Lipusch, N., Ebel, P.: Heading for new shores: Crowdsourcing for en- trepreneurial opportunity creation. In: European Conference of Information Systems (ECIS) (2018) 10. Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017) 11. Elkan, C.: The foundations of cost-sensitive learning. In: Proceedings of the 17th Inter- national Joint Conference on Artiﬁcial Intelligence - Volume 2. pp. 973–978 (2001) 12. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efﬁcient and robust automated machine learning. Advances in Neural Information Processing Systems pp. 2962–2970 (2015) 152 A. Calma, D. Dellermann

13. Gomes, R.G., Welinder, P., Krause, A., Perona, P.: Crowdclustering. Advances in neural information processing systems pp. 558–566 (2011) 14. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016), http://www. deeplearningbook.org [Online; accessed 22-Dec-2018] 15. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks (2014), http://arxiv.org/abs/ 1406.2661, cite arxiv:1406.2661 16. Gregor, S.: Explanations from knowledge-based systems and cooperative problem solving: an empirical study. International Journal of Human-Computer Studies 54(1), 81– 105 (2001) 17. Holzinger, A.: Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Informatics 3(2), 119–131 (2016) 18. Kamar, E.: Directions in hybrid intelligence: Complementing AI systems with human intelligence. In: IJCAI International Joint Conference on Artificial Intelligence (2016) 19. Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., Mullainathan, S.: Human decisions and machine predictions. The quarterly journal of economics 133(1), 237–293 (2017) 20. Lampert, T.A., Stumpf, A., Gancarski, P.: An empirical study into annotator agreement, ground truth estimation, and algorithm evaluation. IEEE Transactions on Image Process- ing 25(6), 2557–2572 (2016) 21. Malone, T.W., Bernstein, M.S.: Collective intelligence handbook (2015) 22. McCullagh, P.: Regression models for ordinal data. Journal of the Royal Statistical So- ciety. Series B (Methodological) 42(2), 109–142 (1980) 23. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human- level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015) 24. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011) 25. Polikar, R.: Ensemble Machine Learning, chap. Ensemble Learning. Springer, Boston, MA (2012) 26. Raileanu, L.E., Stoffel, K.: Theoretical comparison between the gini index and information gain criteria. Annals of Mathematics and Artificial Intelligence 41, 77–93 (2000) 27. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why Should I Trust You?”: Explaining the Pre- dictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, Au- gust 13-17, 2016. pp. 1135–1144 (2016) 28. van Rijn, J.N., Hutter, F.: An empirical study of hyperparameter importance across datasets. In: Proceedings of the International Workshop on Automatic Selection, Config- uration and Composition of Machine Learning Algorithms (AutoML 2017). vol. 1998, pp. 97–104 (2017) 29. Settles, B.: Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin, Department of Computer Science (2009) 30. Shashua, A., Levin, A.: Ranking with large margin principle: Two approaches. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Advances in Neural Information Process- ing Systems 15, pp. 961–968. MIT Press (2003) 11 Decision Support with Hybrid Intelligence 153

31. Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., et al.: Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742 (2017) 32. statista: https://www.statista.com/chart/11690/the-top-reasons-startups-fail/ (2017), [Online; accessed 22-Dec-2018] 33. UnternehmerTUM: https://www.unternehmertum.de/ﬁles/ 7F00000101622FD0DA64373672E58A9C.pdf (2018), [Online; accessed 18-May- 2018]

Part IV

Self-adaptation and Self-learning in Computer Architecture

Improving the Energy Efﬁciency of IoT-Systems and its Software

Norbert Schmitt

Chair of Software Engineering, University of Wurzburg,¨ Wurzburg,¨ Germany, [email protected], Home Page: https://se.informatik.uni-wuerzburg.de

Abstract. The growing amount of small and inexpensive Internet of Things (IoT) devices consumes energy. But also more opportunities are opened up to conserve energy on operating these devices, not only by monitoring the energy demand but by adapting the IoT system itself. Thus improving energy efficiency by moving computations to IoT or edge devices or the cloud that are suited for a given computation at a particular time. This work presents a vision on how such an IoT system can look and the challenges that are inherent to the problem of runtime self-adaption in the context of code offloading to preserve energy, including improving the energy efficiency of software at design-time through informed design choices.

Keywords: Energy Efﬁciency, Code Ofﬂoading, Self-Aware, Cloud, Green Software Engi- neering

12.1 Introduction

In 2011, Cisco estimated a total of 50 billion Internet of Things (IoT) devices operating by 2020 [15]. This number was later scaled down in 2016, with estimations ranging between 20 to 30 billion devices [31]. Even with the more conservative estimations, the amount of IoT devices is still growing rapidly and growth is assumed to continue in the future. All these devices produce data that needs to be processed to be of value. The processing is usually done by cloud data centers. Yet, according to a New York Times study from 20121 data centers already consume large amounts of energy. About 30 billion watts per hour worldwide. The U.S. Environmental Pro- tection Agency estimated in 2010 that the U.S. used 3% of its total electricity to run data centers [27]. With the growing amount of data coming from IoT devices, more

1 https://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of- energy-belying-industry-image.html (last access: 10/02/2019) 158 N. Schmitt data centers would be necessary. Therefore, making IoT systems consisting of many IoT devices and centers more energy efficient becomes more important. Extensive research has been done to make data centers more energy efficient. Rang- ing from the machine level, with more efficient hardware [33], up to intelligently placing applications inside a data center [22]. And although the hardware itself is becoming more energy efficient, there is still little awareness on the impact of software on energy efficiency and the energy efficiency of the software itself. A device’s power consumption is indirectly controlled by the software that is defining what data and how data has to be processed [8]. With code offloading, the software not only decides what and how data has to be processed but also where. Most code offloading techniques are concerned mainly with improving the responsiveness of applications on devices with reduced processing power, like smartphones and other mobile devices. The second major topic for code offloading is conserving battery runtime. Regardless, no code offloading technique considers a bigger picture of many low- power interconnected IoT and edge devices. Thus, possible efficiency improvements by optimising the global state, the time and location where computations take place, instead of single IoT or edge devices and reducing the computational load on the cloud are not considered. Hence, the major goal of this vision is improving the energy efficiency for IoT systems, many connected IoT and edge devices, and software in general. By leveraging edge computing, pushing computations to devices at the edge of a network, like switches and routers, the load on the IoT devices and the cloud can be reduced. Offloading data processing on edge devices should be done with care and only if an effect on the overall energy efficiency of the system, all interconnected devices, can be achieved. Furthermore, not all computations might be feasible for offloading. Depending on a device’s capabilities, like available processing power, in combination with the constraints, like the size of the data to be processed and the time to result (timeliness), it can be beneficial or disadvantageous to use edge devices for computation. Thus this vision intends to employ the self-x attributes of Self-Aware Computing [18, 25] and Organic Computing [39, 30] to increase energy efficiency. By adapting and optimising the current state either by the system itself or controlled by a central entity. Through self-optimisation and self-configuration, the system itself learns and decides autonomously where the data processing should take place, the IoT device, the edge or in the cloud. In case data processing is pushed to the cloud auto-scalers geared toward energy efficiency can help reduce energy consumption further. Auto-scalers can base their decision on energy consumption rather than resource usage. Additionally giving developers the right tools to make them more aware of the energy efficiency of their software and the implications of design decisions can further help in reducing energy consumption. This vision, therefore, proposes new research in code metrics that can represent the energy efficiency of software, starting at existing complexity metrics. This vision proposes a novel way to distribute computations between the IoT devices, edge devices and the cloud as well as novel code metrics to help make developers informed design decisions to increase energy efficiency. 12 Improving the Energy Efficiency of IoT-Systems and its Software 159

The remainder of this paper is structured as follows: First, an overview of the current related work is presented. Section 12.3 describes the current state of the art in energy efficiency in data centers, code offloading and code metrics. In the following Section 12.4, identified challenges and problems are explained together with possible approaches of how to solve these challenges. Finally, this paper concludes in Section 12.5.

12.2 Related Work

Energy efficiency is a growing concern. Therefore extensive research has been done to improve the energy efficiency. The release of the Server Efficiency Rating Tool (SERT) [28] 2.0 in 20172 also demonstrates a consistent interest in this topic. With the growth of data centers for cloud computing and virtualisation, the degrees of freedom to conserve energy have further increased from machine level to whole data centers. This section gives a short overview of the related work, grouped into three topics, cloud energy efficiency, code offloading and software energy efficiency.

12.2.1 Cloud Energy Efﬁciency

The most common reduction in power consumption in cloud computing is through virtual machine (VM) migration [22]. A management software decides on which physical machine a VM gets deployed [20, 41, 38, 3]. They focus on reducing the power consumption without or minimal impact on performance to increase the energy efficiency of cloud servers. Other examples are considering more constraints, such as quality of service (QoS) [2, 3] or service level agreements (SLA) [9]. They all rely on common utilisation power models or build their own model, such as Tian et al. [38]. Yet, servers are not the only power consumer in large data centers. By virtualising formerly physical devices of the network infrastructure, researchers try to improve energy efficiency further [6, 5]. This paper focuses less on cloud energy efficiency and the placement of applications inside a data center, but rather scaling the needed application instances inside the cloud efficiently with an energy-aware auto-scaler that is either reactive [11] or proactive [1].

12.2.2 Code Ofﬂoading

Code ofﬂoading is not a new technology and it has been argued that it can be ben- eﬁcial under certain conditions [26]. Namely, the trade-off between computational and communication effort must be towards low communication and high computation. There is also a variety of frameworks available with different goals [14, 12, 24,

2 https://www.spec.org/sert/SERT-2.0-press-release.html (last access: 10/02/2019) 160 N. Schmitt

17, 16, 34]. Yet, they either focus on scalability of the application, performance or conserving battery capacity for mobile devices. They also can only offload computations to the cloud and not between devices. As only the own individual devices are optimised for saving energy and are limited to cloud offloading, this work proposes an offloading technique that can share work between IoT, edge and cloud devices for the benefit of saving energy in the overall system.

12.2.3 Software Energy Efﬁciency

Software developer choices can influence the energy consumption. The authors show that different implementations for the same task as well as different parameterisation, in the case of Singh et al. the buffer size for reading and writing files, can change the energy consumption without changing the programme’s semantics [36]. Efforts were undertaken to determine the energy consumption resulting from executing a given source code, but most work relies to some degree on information about the hardware. Modeling a software’s energy consumption is either done by training a model with measurements [35] or by defining a workload profile from user input together with expert knowledge of the hardware [19]. This paper tries to find new metrics that can reflect the energy consumption of a running software by source code analysis.

12.3 Foundation

This section gives a short introduction to the definitions and metrics used throughout this paper. This includes energy efficiency, the most efficient state for an IoT system, the cyclomatic complexity and Halstead complexity metrics.

12.3.1 Energy Efﬁciency

This section is divided into two parts. First, the definition of energy efficiency, followed by the definition of the most energy efficient state of an IoT system. The term energy efficiency in this work is used according to Equation 12.1, also used by the Standard Performance Evaluation Corporation (SPEC) [23]. The efficiency eff is the 1 throughput t per watt (power p). The throughput is given in s , the number of finished function calls per second. t 1 eff = (12.1) p Ws

An IoT system in this paper is deﬁned as a network of IoT devices, from sensors to mobile devices, and also including edge devices and the cloud. The structure of the 12 Improving the Energy Efﬁciency of IoT-Systems and its Software 161

A C C

A A B A A A A A A

C C B C C C B B C C B

Dev. 3 Dev. 2 Dev. 1 Dev. 4 Dev. 3 Dev. 2 Dev. 1 Dev. 4 Dev. 3 Dev. 2 Dev. 1

State 1 - Optimal State 2 - Suboptimal State 3 - Optimal

Fig. 12.1: Resuming to an optimal, more energy efficient, state for the overall system after a mobile device (Dev. 4) joins the system. network itself is not relevant in this case and can be a mesh or hierarchy. The optimal state of such an IoT system is considered to be in such a way that not each individual device consumes the least amount of energy for a given computation but the combined energy consumption for all necessary computations in the system. Due to this definition, not each device’s power draw is minimised. The definition allows individual devices to be less efficient if it benefits the overall system. An example is shown in Figure 12.1. Here an IoT system with different computations, A, B and C, is in an optimal state but changes due to Dev. 4 joining the network. As the additional device disturbs the current state, transfering it into a suboptimal state, the system must change to restore a better energy efficiency. The system than uses code offloading on methods A and C on Dev. 4 to return itself to a new optimal state including Dev. 4.

12.3.2 Software Metrics

For preliminary measurements, two well known software metrics were used. The cyclomatic complexity and the Halstead complexity metrics. The cyclomatic complexity represents the programme in the form of a control ﬂow graph. Nodes (N) in the graph represent locations in the code at which a decision must be made. The directed edges E show the ﬂow of the programme after a decision is made. The cyclomatic complexity C is then calculated as follows:

C = E − N + 2P (12.2)

E is the number of edges in the control ﬂow graph and N is the number of nodes. The value P represents the number of connected components of the graph. For the preliminary measurements containing only single functions and no real world programme, connected components P is equal to 1. Thus simplifying the equation to C = E − N + 2. The Halstead complexity metrics build upon four measures. They include: 162 N. Schmitt

• N1: Total number of operators • N2: Total number of operands • η1: Number of distinct operators • η2: Number of distinct operands From these measure, several metrics can be calculated to express a programme’s complexity and size. By adding the distinct and total counters to the vocabulary η = η1 + η2 and length N = N1 + N2 of a programme, several metrics can be calculated shown in Equation 12.3.

η ˆ = η · η + η · η = · η = 1 · N2 N 1 log2 1 2 log2 2 V N log2 D 2 η2 2 E E 3 E = D ·VT= B = (12.3) 18 3000 • Nˆ : The calculated programme length (not lines of code). • V: The programme volume. A second measure, next to Nˆ , of a programme’s size. • D: The difﬁculty to write the programme, how complex is the code. • E: The effort to write the programme, how much work must be put in to produce the code. • T: Calculated time to write the programme, how long it will take to write the code. • B: The number of delivered bugs.

12.4 Challenges

In the following section, the challenges identified in this paper are presented together with first ideas for a solution and preliminary measurements if available. The challenges are divided in runtime challenges and design-time challenges. Runtime challenges include problems arising during operation of an IoT system while the design-time challenges are concerned with developing energy efficient software.

12.4.1 At Runtime

IoT systems consist of a multitude of different devices, as shown in Figure 12.2, each with their capabilities regarding computational power, network, and energy consumption. IoT devices can have very little to no computational power and act just like data sources. Other devices can have more computational power and therefore higher energy consumption but can do a certain amount of data processing themselves without the cloud. The workload that a device can stem is not only constrained by its hardware but also its energy source and computation speed. These constraints can be, but are not limited to: 12 Improving the Energy Efﬁciency of IoT-Systems and its Software 163

Cloud

User

Edge device IoT device with processing capability IoT device without processing capability Mobile IoT device

Fig. 12.2: Internet of Things System Example

• Time and space complexity of the algorithm to compute. • Size of the data set. • Energy consumption per computation (energy efficiency). • Timeliness of the result, including latency. • Locality of data (privacy). For example, a battery powered surveillance camera could provide face recognition itself as long as the battery holds enough charge and offload the face recognition to an edge device or cloud server when the battery runs low. Additionally, devices can be mobile. While stationary devices often communicate with fixed devices after setup, moving devices switch between communication partners frequently to stay available in the network to transmit data. The network itself can be organised in a hierarchical or a mesh structure. Even in the case the system already reached the most energy efficient state, disturbances by a change in the environment, either by a defect, by sensors moving in and out of reach of base stations or by sensors with critically low battery charge, can result in a suboptimal state. For this reason, an automatic offloading of computations to other devices should be triggered. The offloading can be done either by a central instance or by the device that needs offloading if the optimal state is disturbed. The current state developing for IoT devices is, that each device must be programmed and set up independently and is usually limited to this setup. If the programming and deployment are not done by an expert, it is most likely not in an optimal state regarding energy efficiency. The vision, therefore, aims to remove expert knowledge and rely on self-improvement. It is divided into two parts, the first, making and keeping the IoT system more energy efficient by reactively or proactively distributing compu- 164 N. Schmitt tations among IoT devices, edge devices and the cloud. Secondly, improving energy efficiency of the software, primarily computations performed on IoT devices. Under these circumstances, the following challenges were identified.

C1: How can the energy efﬁciency of the system be modelled?

Before the system can make any meaningful decision about itself, it must know its state. Hence a model of the system must be created from which decisions can be derived. The current assumption is that there exist two different decisions that must be made possible by such a model. The first one, is an immediate decision that must not be optimal but quick to react to sudden changes in the environment like a defect. The second decision is a long term planning to optimise the system. An idea is to minimise the amount of immediate decisions and proactively shifting computations by predicted disturbances in the environment. The model can be based on hardware specifics or the running workload. For the hardware specific model, performance counter have proven to be suited well as a multitude of models show [21, 13, 37, 10, 29, 4, 32, 40]. Performance counter have the downside that expert knowledge about a device must be available to be useful. Other higher-level metrics such as CPU usage are also feasible options. For models not dependent on hardware, request rates or similar metrics can act as a basis. Both options should be explored to overcome Challenge C1.

C2: When should the computation be ofﬂoaded from a device?

After determining the current state in Challenge C1, the next challenge can be adressed. For Challenge C2, the idea is to show that each device has certain trade- offs regarding its constrains mentioned earlier. As stated in Section 12.2, offloading might only be beneficial under these constraints. First, a decision algorithm must be derived if code offloading should actually take place, taking into account the battery state (if any), the size of the data to transmit compared to the network bandwidth, the timeliness including possible network latencies and the energy consumption of the devices involved in the offloading process. To derive a decision algorithm, first a careful selection of devices with different characteristics and capacities has to be made to allow for a general answer. Furthermore, not only the hardware is important but also the software. To show that different computations are more suitable to run on edge, IoT devices, or the cloud, a selection of different algorithms with different time and space complexities must be selected as synthetic workloads. For example, different sorting algorithms like randomised quicksort with O (nlog(n)) and inser- 12 Improving the Energy Efficiency of IoT-Systems and its Software 165 tion sort with O n2 for sorting data. Real world workloads should also be included like face recognition. It can then be determined how the energy efficiency behaves in relation to increasing computational complexity, increasing size of data sets and time-to-result.

C3: Where should the computation take place?

As a next step for the Challenge C3, a deployment model must be derived, based on the results of C2 and the state of the art in placement algorithms. This includes knowing what data size is to expected for a specific computation. For all devices in the IoT system, its optimal and maximal computational capacity must be known. While the capacity for a specific device can be read from data sheets or measured, the requirements of the algorithms must be estimated, simulated or modelled, as running them on each device can quickly become infeasible due to the large variety of devices. C3 will be addressed by first deriving a simple decision tree, checking if all requirements of an algorithm are met for deployment on a certain device. If not all requirements are met, the workload cannot be computed on the device in question and another one must be chosen. This can later be extended to more complicated and better solutions if necessary. However, more important is to identify the resource and time-to-result requirements of the algorithm. It is intended to use static code analysis combined with machine learning to estimate the resource requirements. The time- to-result requirements must be provided by the developer as identical algorithms can have different requirements.

C4: How to design an auto-scaler to make informed decisions about energy efﬁciency?

Auto-scalers are used in cloud environments to automatically reduce or increase the number of resources, normally virtual machines or container. This is mostly used to save money as only the necessary amount resources are used and paid for. An auto- scaler could also be used to conserve energy by deploying, shutting down and moving application instances. By giving the auto-scaler a model of the IoT system designed in Challenge C1 it can make informed decisions about energy efﬁciency. However, solving a model can take large amounts of time, especially for IoT and edge devices with limited computational capacity. Table 12.1 shows the time to solve performance models of increasing size. As the typical cloud is used by many customers, solving a model for each customer to determine the best solution is time consuming. A cloud 166 N. Schmitt

Application Simulation Time (s) Media Store 25.6 SPECjEnterprise2010 55.6 Process Control System 65.4 Business Reporting System 587.6

Table 12.1: Solving time for different performance models from Brosig et al. [7]. with 1000 customers, in this case IoT systems, and an average solving time of 30s needs over 8 hours to solve all models, not including the time to reach a decision from the model’s output.

12.4.2 At Design-Time

With the previous challenges focusing on the runtime aspect, the remaining challenges focus on the design time of the system. To improve energy efficiency further, the developers should be more aware if and how certain design decisions and programming patterns can influence energy efficiency. To be helpful for the developer, a quick response is key or otherwise the information will not be integrated in the final programme. If a developer must wait several minutes or even hours to know the impact of a small code change will discourage its usage. With the added complexity that the information should be as hardware independent as possible, the energy efficiency must be estimated without actually benchmarking the code on real hardware. Re- quirements on the system can change over time. While the devices can be exchanged and the system should adapt itself, the software on the other hand must be adapted manually. By changing the software, it can direct how energy is consumed on a device without changing the software’s semantics. Determining how energy efficient a piece of software can run from the sources alone is challenging.

C5: Are current metrics suitable to determine the runtime energy efﬁciency behaviour of software?

To overcome Challenge C5, first, it must be surveyed at what level design decisions can be made that negatively affect energy efficiency. This could include design patterns, programming language keywords and others. Their influence on energy efficiency must be identified to give valuable feedback to the developer. The first idea is to check existing code metrics and how they relate to energy efficiency. As a first impression, seven algorithms with different runtime behaviours O n2 , O (n · log(n)), O (n) and O (log(n)) were selected and measured. The algorithms are: 12 Improving the Energy Efficiency of IoT-Systems and its Software 167

• Linear search • Binary search • Mandelbrot escape time algorithm • Sieve of Eratosthenes (prime number calculation) • Insertion Sort • Merge Sort • Quick Sort All were measured with data sizes ranging from 400 to 4096·102 bytes. The size was doubled after each measurement of 240 seconds, resulting in 11 total measurements for each algorithm per measurement run. The only exception is the Mandelbrot escape time algorithm with only 9 measurements per measurement run. For the escape time algorithm, image sizes starting at 100x100 were calculated. First the x value is doubled. Afterwards the y value is doubled up to the maximum image size of 1600x1600. Larger image sizes were not applicable. Each measurement run is repeated eight times. From the data, the average bytes processed per Ws is calculated with Equation 12.4 with n = 8. The number of iterations (how often the algorithm could be executed in a measurement) after 240s is recorded to calculate the number of bytes processed, or generated in the case of the escape time algorithm, by multiplying it with the known data size for each iteration. The idle power pidle is subtracted from the measured average power pavg to only show the power consumed by the active application.

n · 1 · iterationsi bytesPerIterationi ∑ ( − ) · (12.4) n i=1 pavg pidle 240s

From the measurement data, the average bytes processed per Ws is presented in Figure 12.3 to show how energy efficient the implementation is. The calculated complexity metric is on the x-axis and average bytes processed is on the y-axis of Fig- ure 12.3. It seems odd that insertion sort has better energy efficiency in this scenario then the divide-and-conquer sorting algorithms. This behaviour could be due to the linear iteration over the data array making better use of the optimisations in hardware like memory prefetchers. The figure also shows two groups of algorithms. Group 1 with the linear search, binary search and insertion sort, group 2 with the sieve of Eratosthenes, merge and quick sort and mandelbrot escape time algorithm as a possible outlier. Table 12.2 shows the Pearson correlation after applying the log function to both groups. The measurement hints that arranging algorithms into groups can give reasonably accurate results with existing code metrics to represent energy efficiency of an implementation. While the correlation is very high, the values should be taken with skepticism as only three data points are available for each group and further measurements must be made to support the first impression. Especially as the correlation including all 168 N. Schmitt

109 109 109

108 108 108

107 107 107

106 106 106 bytes / Ws 5 5 5 10 Linear Search 10 10 Binary Search Mandelbrot 4 Sieve of Eratosthenes 4 4 10 Insertion Sort 10 10 Merge Sort Quick Sort 103 103 103 02468 0 1020300 50 100 150 200 Cyclomatic Complexity Lines of Code Calculated Program Length

Fig. 12.3: Static code metrics and the amount of data (bytes) processed for each Ws consumed by different algorithms.

Metric C LoC η N NVDˆ Correlation All −0.1996 −0.5142 −0.5868 −0.3832 −0.5888 −0.4263 −0.0293 Group 1 0.9925 0.9971 0.9996 0.9944 0.9989 0.9923 0.9957 Group 2 0.9484 0.9982 0.7185 0.9562 0.6820 0.9360 0.9534

Table 12.2: Correlation coefﬁcients for different algorithms and software metrics. algorithms does not support the use of existing complexity code metrics, shown in Section 12.3.2. The preliminary measurements show that code metrics can possibly be used to give developers a base for comparison of different implementations but further work is necessary to identify the code constructs that are responsible for making an algorithm run energy efﬁcient or not.

C6: How can the energy efﬁciency behaviour of the source code be communicated to the developer?

As a second part to convey energy efficiency of software to developers is the introduction of a energy efficiency regression test, similar to unit testing. As DevOps is growing among software developers, the time between a new version of software and deployment is reducing. To ensure that energy efficiency remains constant or improves, developers need to be aware of the implications of changes in the source code to the energy efficiency. Based on the results from Challenge C5, a regression test could be implemented to show if the energy efficiency of the software has improved 12 Improving the Energy Efficiency of IoT-Systems and its Software 169 over time. While the metrics in C5 can be calculated on the developers machine to provide quick feedback, a regression test can run predefined workloads on the application on a reference machine. Although this machine could possibly be a virtual machine, it is highly depending on the metrics, as some metrics could be dependent on hardware information such as performance counter that are not available in a virtualised environment.

12.5 Conclusion and Future Work

This paper introduces the vision of making IoT systems and their software more energy efficient. Six high-level challenges in Section 12.4 are presented that need to be overcome. The challenges include finding suitable modelling techniques, monitoring parameters for code offloading between devices and application placement, cloud auto-scaling. Additionally, static code metrics representing energy efficiency are discussed, making developers more aware of energy efficiency in their software design choices. Ideas and possible solutions, where to execute certain computations in an IoT system, are presented. The preliminary measurements for static code metrics show that they might be usable if algorithms or even full applications can be sorted into groups with different energy efficiency behaviour. As a first step towards more energy efficient IoT systems, an abstract model representation according to Challenge C1 is vital. After the state of an IoT system can be determined, progress in the following runtime challenges can be achieved. In Challenge C3, after building a decision model, with a decision tree as a first choice, current suitable solutions to the placement problem must be surveyed and evaluated. The design time challenges in Section 12.4.2 can be addressed simultaneously. First deriving a new static code metric to allow comparison of semantically identical piece of source code and secondly, given developers a tool to raise awareness that design time decisions can impact energy efficiency. This could be achieved with a energy efficiency regression testing or other options.

References

1. Bauer, A., Herbst, N., Spinner, S., Ali-Eldin, A., Kounev, S.: Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field. IEEE Transactions on Par- allel and Distributed Systems (September 2018), preprint 2. Beloglazov, A., Abawajy, J., Buyya, R.: Energy-aware resource allocation heuristics for efﬁcient management of data centers for cloud computing. Future generation computer systems 28(5), 755–768 (2012) 3. Beloglazov, A., Buyya, R.: Energy Efﬁcient Resource Management in Virtualized Cloud Data Centers. In: 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. IEEE (2010) 170 N. Schmitt

4. Bircher, W.L., John, L.K.: Complete system power estimation using processor performance events. IEEE Transactions on Computers 61(4), 563–577 (April 2012) 5. Bolla, R., Bruschi, R., Lombardo, C., Mangialardi, S.: DROPv2: Energy-Efficiency through Network Function Virtualization. IEEE Network 28(2), 26–32 (Apr 2014) 6. Botero, J.F., Hesselbach, X., Duelli, M., Schlosser, D., Fischer, A., de Meer, H.: Energy Efficient Virtual Network Embedding. IEEE Communications Letters 16(5), 756–759 (Mar 2012) 7. Brosig, F., Meier, P., Becker, S., Koziolek, A., Koziolek, H., Kounev, S.: Quantita- tive Evaluation of Model-Driven Performance Analysis and Simulation of Component- based Architectures. IEEE Transactions on Software Engineering (TSE) 41(2), 157–175 (February 2015), http://dx.doi.org/10.1109/TSE.2014.2362755 8. Calero, C., Piattini, M.: Green in software engineering. Springer (2016) 9. Chase, J.S., Anderson, D.C., Thakar, P.N., Vahdat, A.M., Doyle, R.P.: Managing energy and server resources in hosting centers. ACM SIGOPS operating systems review 35(5), 103–116 (2001) 10. Chen, X., Xu, C., Dick, R.P., Mao, Z.M.: Performance and power modeling in a multi- programmed multi-core environment. In: Proceedings of the 47th Design Automation Conference. pp. 813–818. DAC ’10, ACM, New York, NY, USA (2010), http://doi.acm. org/10.1145/1837274.1837479 11. Chieu, T., more: Dynamic Scaling of Web Applications in a Virtualized Cloud Compu- ting Environment. In: IEEE ICEBE 2009. pp. 281–286. IEEE (2009) 12. Chun, B.G., Ihm, S., Maniatis, P., Naik, M., Patti, A.: Clonecloud: elastic execution between mobile device and cloud. In: Proceedings of the sixth conference on Computer systems. pp. 301–314. ACM (2011) 13. Contreras, G., Martonosi, M.: Power prediction for intel xscale/spl reg/ processors using performance monitoring unit events. In: ISLPED ’05. Proceedings of the 2005 Inter- national Symposium on Low Power Electronics and Design, 2005. pp. 221–226 (Aug 2005) 14. Cuervo, E., Balasubramanian, A., Cho, D.k., Wolman, A., Saroiu, S., Chandra, R., Bahl, P.: Maui: making smartphones last longer with code offload. In: Proceedings of the 8th international conference on Mobile systems, applications, and services. pp. 49–62. ACM (2010) 15. Evans, D.: The Internet of Things - How the Next Evolution of the Internet Is Changing Everything. Tech. rep., Cisco Internet Business Solutions Group (IBSG) (April 2011), white Paper 16. Flores, H., Srirama, S.: Adaptive code offloading for mobile cloud applications: Exploit- ing fuzzy sets and evidence-based learning. In: Proceeding of the fourth ACM workshop on Mobile cloud computing and services. pp. 9–16. ACM (2013) 17. Gordon, M.S., Jamshidi, D.A., Mahlke, S.A., Mao, Z.M., Chen, X.: Comet: Code offload by migrating execution transparently. In: OSDI. vol. 12, pp. 93–106 (2012) 18. Hahner,¨ J., von Mammen, S., Timpf, S., Tomforde, S., Sick, B., Geihs, K., Stumme, G., Hornung, G., Goeble, T.: ?Know Thyselves – Computational Self-Reflection in Collec- tive Technical Systems. In: Proceedings of the 29th GI/ITG International Conference on Architecture of Computing Systems ? ARCS Workshops, held 4 ? 7 April 2016 in Nuremberg, Germany, Workshop on Self-Optimisation in Autonomic and Organic Com- puting Systems (SAOS16). pp.1–8.VDEVerlag (2016) 19. Hao, S., Li, D., Halfond, W.G., Govindan, R.: Estimating mobile application energy consumption using program analysis. In: Proceedings of the 2013 International Conference on Software Engineering. pp. 92–101. IEEE Press (2013) 12 Improving the Energy Efficiency of IoT-Systems and its Software 171

20. Hiltunen, M.A., Schlichting, R.D., Jung, G., Pu, C., Joshi, K.R.: Mistral: Dynami- cally managing power, performance, and adaptation cost in cloud infrastructures. In: 2010 IEEE 30th International Conference on Distributed Computing Systems(ICDCS). vol. 00, pp. 62–73 (06 2010), doi.ieeecomputersociety.org/10.1109/ICDCS.2010.88 21. Isci, C., Martonosi, M.: Runtime power monitoring in high-end processors: Methodol- ogy and empirical data. In: Proceedings of the 36th Annual IEEE/ACM International Symposium on Microarchitecture. pp. 93–. MICRO 36, IEEE Computer Society, Wash- ington, DC, USA (2003), http://dl.acm.org/citation.cfm?id=956417.956567 22. Jin, Y., Wen, Y., Chen, Q.: Energy Efficiency and Server Virtualization in Data Centers: An Empirical Investigation. In: 2012 IEEE Conference on Computer Communications Workshops. pp. 133–138 (Mar 2012) 23. von Kistowski, J., Lange, K.D., Arnold, J.A., Sharma, S., Pais, J., Block, H.: Measuring and benchmarking power consumption and energy efficiency. In: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering. pp. 57–65. ICPE ’18, ACM, New York, NY, USA (2018), http://doi.acm.org/10.1145/3185768.3185775 24. Kosta, S., Aucinas, A., Hui, P., Mortier, R., Zhang, X.: Thinkair: Dynamic resource allocation and parallel execution in the cloud for mobile code offloading. In: Infocom, 2012 Proceedings IEEE. pp. 945–953. IEEE (2012) 25. Kounev, S., Kephart, J.O., Milenkoski, A., Zhu, X. (eds.): Self-Aware Computing Sys- tems. Springer Verlag, Berlin Heidelberg, Germany (2017), http://www.springer.com/ de/book/9783319474724 26. Kumar, K., Lu, Y.H.: Cloud computing for mobile users: Can offloading computation save energy? Computer 43(4), 51–56 (2010) 27. Lange, K.D., Tricker, M.G.: The Design and Development of the Server Efficiency Rat- ing Tool (SERT). In: Proceedings of the 2nd ACM/SPEC International Conference on Performance Engineering. pp. 145–150. ICPE ’11, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/1958746.1958769 28. Lange, K.D., Tricker, M.G.: The Design and Development of the Server Efficiency Rat- ing Tool (SERT). In: ICPE 2016. pp. 145–150. ICPE ’11, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/1958746.1958769 29. Lim, M.Y., Porterfield, A., Fowler, R.: Softpower: Fine-grain power estimations using performance counters. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing. pp. 308–311. HPDC ’10, ACM, New York, NY, USA (2010), http://doi.acm.org/10.1145/1851476.1851517 30. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 31. Nordrum, A.: The Internet of fewer things [News]. IEEE Spectrum 53(10), 12–13 (2016) 32. Rodrigues, R., Annamalai, A., Koren, I., Kundu, S.: A study on the use of performance counters to estimate power in microprocessors. IEEE Transactions on Circuits and Sys- tems II: Express Briefs 60(12), 882–886 (Dec 2013) 33. Rotem, E., Naveh, A., Ananthakrishnan, A., Weissmann, E., Rajwan, D.: Power- management architecture of the intel microarchitecture code-named sandy bridge. IEEE Micro 32(2), 20–27 (March 2012) 34. Schafer, D., Edinger, J., Paluska, J.M., VanSyckel, S., Becker, C.: Tasklets:” better than best-effort” computing. In: Computer Communication and Networks (ICCCN), 2016 25th International Conference on. pp. 1–11. IEEE (2016) 35. Seo, C., Malek, S., Medvidovic, N.: Component-level energy consumption estimation for distributed java-based software systems. In: International Symposium on Component-Based Software Engineering. pp. 97–113. Springer (2008) 172 N. Schmitt

36. Singh, J., Naik, K., Mahinthan, V.: Impact of developer choices on energy consumption of software on servers. Procedia Computer Science 62, 385–394 (2015) 37. Singh, K., Bhadauria, M., McKee, S.A.: Real time power estimation and thread scheduling via performance counters. SIGARCH Comput. Archit. News 37(2), 46–55 (Jul 2009), http://doi.acm.org/10.1145/1577129.1577137 38. Tian, Y., Lin, C., Yao, M.: Modeling and analyzing power management policies in server farms using stochastic petri nets. In: Future Energy Systems: Where Energy, Computing and Communication Meet (e-Energy), 2012 Third International Conference on. pp. 1–9 (May 2012) 39. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 40. Tsafack Chetsa, G.L., Lefevre,` L., Pierson, J.M., Stolf, P., Da Costa, G.: Exploiting performance counters to predict and improve energy performance of HPC systems. Fu- ture Generation Computer Systems vol. 36, pp. 287–298 (Jul 2014), https://hal.archives- ouvertes.fr/hal-01123831 41. Urgaonkar, R., Kozat, U., Igarashi, K., Neely, M.: Dynamic resource allocation and power management in virtualized data centers. In: Network Operations and Manage- ment Symposium (NOMS), 2010 IEEE. pp. 479–486 (April 2010) 13

Self-Aware MPSoC Architecture Extensions for Automotive Applications

Armin Sadighi, Thomas Wild, and Andreas Herkersdorf

Chair of Integrated Systems, Technische Universitat¨ Munchen,¨ Munchen,¨ Germany {armin.sadighi,thomas.wild,herkersdorf}@tum.de

Abstract. With decreasing transistor size and reaching 10-nanometer technologies, the reliability of electronic circuits has reduced. Moreover, the environment, application, and manufacturing variations of today’s applications make the design of Multiprocessor System-on-Chips (MPSoCs) complex. Runtime adaptations are required to cope with all the dynamics, which are almost impossible to deal with at design time. The combination of component-specific, uncorrelated control instances that the state-of-the- art solutions are providing is inadequate to handle multi-criterial objective functions of complex systems. Furthermore, a strictly centralised controller for such systems will not have the scalability to support future systems and hence will not be suitable. Therefore, a viable solution should consist of an implementation that is as modular and distributed as possible, but also centralised wherever necessary to be cost-efficient as well as overhead-efficient and scalable. This paper outlines a solution by introducing a new design paradigm called Information Pro- cessing Factory. The goal is to combine the benefits of self-aware learning systems with formal reactive guarantees to enable MPSoC designs for safety-critical and fail-operational applications.

13.1 Introduction

Multiprocessor System-on-Chips (MPSoCs) are a class of Very-Large-Scale Inte- grated (VLSI) systems that incorporate most or all Intellectual Property (IP) components necessary for an application. These systems have emerged in the past decade and gained popularity since. They are widely used for diverse applications, such as multimedia computing, embedded signal processing, and also server/desktop computing. Continuous scaling of transistors according to Moore’s law has enabled the integration of more and more transistors on a ﬁxed chip area. For MPSoCs these led to increased number of cores and resources that can be integrated in the system, increasing size and complexity of these systems[4]. 174 A. Sadighi et al.

Significant performance can get lost when ignoring aspects, such as memory ac- cesses, synchronisations, and Network on Chip(NoC) interconnects, which increase the complexity of the assessment substantially. Moreover, with highly dynamic operation environments of today’s applications, such as autonomous driving, the need for optimising these systems both in performance and power as well as reliability and security arises, which leads to a multi-dimensional optimisation problem[18]. Reaching nanometer technologies accompanied by an increased number of cores and resources used in MPSoCs brings about some challenges as follows: • Varying performance, power, and reliability parameters not only in different parts of the chip, but also in the same parts across their usage time. • Difficulties in sustaining performance with an unknown and dynamic workload distribution. • Increased vulnerability to environmental conditions. • Extremely high power densities, which lead to thermal hotspots that manifest as aging and wear-outs, further decrease reliability. This document consists of three sections, Motivation and Related Work, Information Processing Factory paradigm, and Objectives of the Dissertation. In Motivation and Related Work, an application domain in which MPSoC systems will be dominant in the future is introduced. The specific problem of using MPSoC in that domain and the current endeavors to find a solution to this problem is briefly investigated in this section. In Information Processing Factory Paradigm, this metaphor and its usage in designing microelectronic devices is explained. Finally, in Objectives of the dissertation, the planned work for the Ph.D. dissertation, with respect to the motivation, is described.

13.2 Motivation and Related Work

Safety-critical systems are systems whose failure could result in loss of life, significant property damage, or damage to the environment. Examples of systems in this domain are automotive control systems, medical devices, industrial control systems, and aviation control systems. Modern safety-critical automotive applications have high performance requirements that can often no longer be met by single-core systems. This holds especially true for future fully autonomous driving applications. Furthermore, the current trend is to consolidate different applications on a single MPSoC to reduce the overall power consumption and hardware cost. This brings the problem of different applications, usually of different criticality levels, competing for shared resources such as memory, I/O ports, and the interconnect. In such a system, it is vital that the safety-critical applications continue their work even in case of an error and, furthermore, cannot be affected by other applications. 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications 175

MPSoCs have great potential to be used in safety-critical embedded systems such as automotive industry. MPSoCs bring the following beneﬁts to safety-critical applications [16]: • Energy and area efﬁciency • Higher computational performance • Special processing capabilities through heterogeneity • Reduction of physical units Despite all these advantages, MPSoCs cannot yet be successfully used in this domain. Their architecture is not focused on real-time safety-critical and fail-operational concepts. The focus of this dissertation will be to use these SoCs in automotive applications with self-awareness and self-organisation properties. To enable the use of MPSoC designs for safety-critical and fail-operational scenarios several research programmes, such as the NSF Variability Expeditions[8] or SPP Dependable Computing[12], developed new solutions ranging from the circuit level all the way up to the software and application layers. The proposed solutions in these research programmes are not only applied at design time, but they also can be applied at runtime, as well as down time of the system like maintenance, and also a combination of all the different situations of the system. For this end, various sensors are used on the chip, that are called on-chip sensors, to monitor the status of the hardware and assess its health situation. Examples for this can be the measurement of the chip temperature to automatically control it and hence mitigate aging effects. Another focus of these programmes was to propose different types of redundancy to guarantee sustainable performance of the system at all times, as well as real-time constraints and integrity for safety critical, and high availability embedded and cyber-physical applications. These mechanisms are mainly uncorrelated and targeted to control individual effects. These solutions are acceptable and state-of-the-art for todays circuits, but as the number and impact of effects are growing, the demand for better solutions to control all the parameters to ensure dependability, resilience, and longevity grows. On the other hand, to control the complexity of MPSoC designs and cope with the environment, application, and manufacturing variations new paradigms like autonomic computing [13], organic computing, and proactive computing [19] were introduced. The goal of these projects was to integrate self-x functions, where x may stand for - control, -organisation, -awareness, etc., in order to attain higher degrees of autonomy in these systems. The idea behind it is to make design decisions at runtime instead of design time to react on dynamics, which are unknown at design time, and allow for optimising/managing the system state. The Organic Computing initiative [14, 20] has paved the path for developing and advancing self-organising systems engineering in Germany. A similar observer/controller-based approach controlling emergent behaviour is adopted in the EU-FP7 EPiCS (Engineering Proprioception in Computing Systems) project. It exploits self-awareness and self-expression in the compute platform, middleware software, distributed network infrastructure as well as application programmes [1]. 176 A. Sadighi et al.

Within Another project focusing on ”Bio-Inspired Adaptive Architectures and Sys- tems” called EPSRC, Dahir et al. from University of York investigated the benefits of self-optimisation to gracefully improve power-efficiency and performance of many- core systems [6]. These systems are thus able to cope with permanent and transient faults and continually seek to attain a more optimal system configuration by using a set of online optimisation mechanisms on spare processing cores. Another work in self-aware architectures was contributed by Ipek and Martinez [3] where artificial neural network machine learning techniques were used to predict system performance as a function of resource allocation decisions at runtime. The SElf-AwarE Computing (SEEC) framework from MIT [11, 10] attempts to reduce the programming effort of MPSoCs with a self-aware programming model. Applications explicitly specify system goals (for power and performance) and a unified decision engine attempts to adapt algorithms, software, and hardware to op- timally meet the set goals. One of the central concepts of SEEC is the Observe- Decide-Act control cycle that is executed in cyclic time intervals in which a certain amount of application-specific work has to be accomplished. SEEC uses on-chip self-monitoring to guarantee predictable behaviour of power-aware systems [9]. The CPSoC (Cyber-Physical System on Chip) project from UC Irvine [17] is an exemplary sensor-rich many-core computing platform that couples on-chip and cross- layer physical and virtual sensing and actuation applied across different layers of the hardware/software system stack to adaptively achieve desired objectives and Quality-of-Service (QoS) [8]. The CPSoC platform takes advantage of an Adaptive- Reflective Middleware which sits between applications and the operating system kernel. The middleware implements closed-loop resource management policies. The middleware offers scalable and autonomous resource management by leveraging formal supervisory control theory combining the strengths of classical control theory with state-of-the-art heuristic approaches to efficiently achieve changing run-time goals [15]. The ASoC (Autonomic System on Chip) project [2]within the “Organic Computing“ investigated the application of hardware and software reinforcement machine learning techniques (learning classifier systems) [5] on homogeneous multicore processors for optimising workload balancing, power consumption, and resilience against intermittent core failures. Self-organised RISC core behaviour was accomplished through hardware-centric reinforcement-based learning classifier tables (LCTs) [21]. The mentioned methods, while optimising the operation point for different scenarios in an MPSoC design are not meant for safety-critical applications. Therefore, the goal is to combine the benefits of self-aware learning designs with formal reactive guarantees in order to enable MPSoC designs for safety-critical and fail-operational applications. In the next section, a new paradigm called Information Processing Factory is introduced that will solve the shortcomings as mentioned above. 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications 177

Fig. 13.1: The IP Factory

13.3 Information Processing Factory Paradigm

To overcome the aforementioned limitations, a new design paradigm called the In- formation Processing Factory is introduced [7]. This paradigm uses self-awareness and self-optimisation to overcome the limits that prevent MPSoCs from being used in the safety-critical domain of automotive. Future microelectronic systems can be compared to factories. To keep factory production running effectively and efﬁciently, the production itself, the logistics of supply, the machinery and transport, and the facility control including heating or illumi- nation are all adapted to the current workload, while considering maintenance and continuous operation. Future microelectronic systems face comparable requirements. There will be a platform operation layer that controls the performance and health status of a microelectronic system based on an on-chip sensor network, with respect to temperature, energy, aging, reliability, security, and long-term systems evolution. This platform operation layer will not only focus on the current status but also predict the future state of a microelectronic system including the expected development of the platform. Such holistically controlled autonomous microelectronic systems can be considered Information Processing Factories (IP Factories), represented in Fig.13.1. Both platform uncertainty and system dynamics are considerable challenges to reach guarantees for real-time safety-critical systems. Platform uncertainty arises from variability in semiconductor production, long-term aging effects, and imprecise models used at design time. System dynamics can arise from platform dynamics, such as temperature variation, from short-term application load dynamics, or from long- term application software change and evolution. Instead of using a single operating point determined at design time, this paradigm suggests employing run-time platform adaptation by taking a new approach to control platform dynamics that com- 178 A. Sadighi et al.

Fig. 13.2: IPF hardware/software stacking bines self-organising machine learning techniques with formal reactive methods to provide worst-case real-time and safety guarantees. IPF is a transatlantic joint collaboration between Technical University Munich, Tech- nical University Braunschweig, and the University of California at Irvine. Each partner has specialised and worked in different layers of abstraction and has gained knowledge and experience in the corresponding layer. In IPF these layers will be brought together in a collaborative and constructive manner to build a holistic autonomic entity that works and shares information in order to make decisions at runtime to fulfill the requirements of the modern MPSoC designs. Also because of the modularity of the layers, they will be evolved independently to become more performant and reliable in order to be able to solve the faced challenges of mixed-criticality applications. Fig.13.21 shows the final envisioned hardware/software stacking of the different IPF layers involved. The bottom MPSoC layer follows a tiled many-core architecture template where heterogeneous compute nodes, denoted by C in the figure, with local on-chip memories, denoted by Mem in the figure, are interconnected via a meshed Network-on-Chip (NoC). The next layer in this hierarchy carries the LCT and a resource manager in the NoC, called RM. These lower two layers of the IPF stack are dominantly realised by hardware building blocks that extend the normal capabilities of available SoC IPs in such a way that no complete redesign is necessary and the state-of-the art designs can also be reused as much as possible. The following two layers, the Reflective IPF Middleware and the Cognitive Applications layers.

1 This ﬁgure is a courtesy of our IPF project partner at TU Braunschweig. 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications 179

In the next section, the objective of this Ph.D. dissertation and some initial ideas are discussed that are currently under investigation.

13.4 Objectives of the Dissertation

The proposed Ph.D. dissertation investigates architectures and methods for the IP factory (IPF) approach in order to achieve cross-layer multi-criterial performance, power, real-time behaviour, and reliability optimisation for application in automotive embedded systems. This work is performed within the BMBF ARAMIS II project. ARAMIS II is a project that investigates the use of multi-core and many-core designs for mixed-critical applications in the domains of automotive and avionics as well as the future topic Industry 4.0. These domains show a clear demand for a continuous increase of processing capabilities. This demand for processing power can be seen for example in highly automated driving and connected machines with real-time requirements. Furthermore, this demand is substantiated by an increasing interaction and integration with other systems and services. This requires the usage of multicore technology in embedded systems in the near future and at the same time the the challenges associated with multi and many core systems have to be solved for these domains. The ARAMIS II project consist of a consortium of 33 partners in Germany, with some leading companies in each mentioned domain e.g. Audi in automotive, Airbus in avionics, and Siemens in industry automation. One of the challenges is to use the reinforcement-based learning for mixed-criticality applications. To enable this a possible extension is to combine the LCT tables, from the ASoC project, with the middleware of the CPSoC architecture to provide a hierarchical learning environment with goal-oriented supervisory control. A good starting point is to treat the critical and best-effort tasks separately. The best-effort tasks can be adapted at runtime based on the learning which happens in the LCTs. The occurrence of a problem is not catastrophic for these tasks. However, critical tasks cannot be adapted without being able to guarantee the correct functionality at runtime. Currently, in mixed-critical systems there is a complete isolation between best-effort tasks and critical tasks. This isolation prevents the access of all the best-effort task to resource that are allocated for the critical tasks. In future designs providing a guarantee for such a complete isolation is both expensive and has a huge design effort. In this research work we want to investigate an innovative approach that does not isolate the different task classes. The chosen approach optimises the system resources for the critical tasks indirectly. One way to achieve this goal is to have monitors that can be used as input to the LCTs for the best-effort tasks, informing them about the critical task. The slack between the deadline for a critical task and its actual ﬁnishing time is chosen as a monitor. If this slack time approaches a certain threshold, the LCTs for the best-effort task will learn that the critical tasks will need more resources and hence limit the best- effort tasks and free more resources for the critical tasks. Fig. 13.3 illustrates this 180 A. Sadighi et al.

(a)

(b)

Fig. 13.3: MPSoC Design with LCT tables: (a) Finish time is close to deadline; (b) Adaptation on best-effort tasks increases the slack concept. In Fig. 13.3(a) the actual finishing time of the critical task is approaching the deadline more than desired. In this case, the LCT will be informed and adaptation will happen on the best-effort tasks, as seen in Fig. 13.3(b). The desired outcome is a safe distance between the finishing time of the critical task and its deadline. This information is provided by the reflective middleware from the CPSoC project. Hence, this is also a starting point for communication and information sharing between the different layers of the IPF platform. For demonstrating the benefits of this approach the scenario that will be run on the system is as follows: the MPSoC system, consisting of LEON3-based cores and a memory system, executes high-critical tasks, such as a pedestrian detection application, which has a tight deadline. Additionally, best effort tasks are executed. Due to problems in the system, like increased interference of the mixed critical tasks on 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications 181 each other, the end times of the critical task are delayed and get closer to the deadline. If this problem is not solved, it might cause a miss of the deadline and hence, a system failure. Therefore, the proposed mechanism should be able to monitor the slack time between the time that the task execution finishes and its deadline. When the measured slack starts to decrease, the system must act by changing the workload, frequency, and utilisation of the best effort tasks to reduce the interference. In this way, all the needed resources for the critical tasks are indirectly provided and their failures are prevented. This solution will be implemented and evaluated on an FPGA platform. Another important aspect of the ongoing work is to provide the LCTs with the ability to dynamically adapt to different goals and conditions based on the provided scenario at operation time and hence, empowering the system with autonomy. As an example, the system might be in a power saving mode and requires to be as power efficient as possible. In this case, the LCT should be able to prioritise rules that meet this requirement. Similarly, when the system requires its full performance in another operation time, the LCT should be able to learn this goal and act upon. Hence, higher layers can learn the system goals based on the information that they obtain from the system. These goals are communicated to the LCTs. Therefore, it is required that the learning layers effectively communicate to each other at suitable times. To achieve this goal, we will investigate methods and models that enable constructive collaboration between the learning layers, which ideally will lead to a suitable emergent behaviour.

13.5 Conclusion

In future MPSoCs, dealing with the changes and evolution of the running application as well as hardware reconﬁguration, also during runtime, are going to be major challenges to conquer complexity. These concepts must be addressed as key challenges in design methodologies for autonomous systems. The existing dilemma is that including all possible uncertainties and dynamics of a processing platform and its operational environment at design time is either impossible or leads to unacceptable designs. The IPF paradigm described in this paper presents a possible solution for the aforementioned dilemma by introducing a hierarchical approach across multiple hardware/software abstraction layers. This approach is highly scalable and modular and can be incorporated in different applications. Furthermore, some early promising approaches for tackling the research problems in this project have been described. 182 A. Sadighi et al. Acknowledgment

This project is partially funded by the DFG Grant HE4584/7-1 as well as the German BMBF project ARAMIS II with funding ID 01 IS 16025. We particularly thank our IPF project partners at TU Braunschweig (Prof. R. Ernst), UC Irvine (Prof. Dutt, Prof. F. Kurdahi and Dr. A. Rahmani) for the excellent collaboration, as well as J. Zeppenfeld and A. Bernauer for their contributions to the predecessor ASoC project.

References

1. Becker, T., et al.: Epics: Engineering proprioception in computing systems. In: IEEE 15th International Conference on Computational Science and Engineering (2012) 2. Bernauer, A., et al.: Generic self-adaptation to reduce design effort for system-on-chip. In: SASO (2009) 3. Bitirgen, R., et al.: Coordinated management of multiple interacting resources in chip multiprocessors: A machine learning approach. In: MICRO (2008) 4. Borkar, S.: Thousand Core Chips-A Technology Perspective. In: 2007 44th ACM/IEEE Design Automation Conference (2007) 5. Butz, M.V.: Rule-Based Evolutionary Online Learning Systems. Springer, ISBN 978-3- 540-25379-2 (2006) 6. Dahir, N., Campos, P., Tempesti, G., Trefzer, M., Tyrrell, A.: Characterisation of feasibility regions in fpgas under adaptive dvfs. In: FPL. pp. 1–4 (Sept 2015) 7. Dutt, N., et al.: Conquering mpsoc complexity with principles of a self-aware information processing factory. In: CODES+ISSS (2016) 8. Gupta, P., et al.: Underdesigned and Opportunistic Computing in Presence of Hardware Variability. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2013) 9. Hoffmann, H.: Coadapt: Predictable behavior for accuracy-aware applications running on power-aware systems. In: 2014 26th Euromicro Conference on Real-Time Systems. pp. 223–232 (July 2014) 10. Hoffmann, H., et al.: Application heartbeats: A generic interface for specifying program performance and goals in autonomous computing environments. In: ICAC. pp. 79–88. ICAC ’10, ACM (2010) 11. Hoffmann, H., et al.: Eec: A framework for self-aware computing. Tech. rep., Mas- sachusetts Institute of Technology, Cambridge, (oct 2010) 12. http://spp1500.itec.kit.edu/index.php: 13. Kephart, J.O., et al.: The vision of autonomic computing. Computer 36(1), 41–50 (Jan 2003) 14. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 15. Rahmani, A., et al.: SPECTR: Formal Supervisory Control and Coordination for Many- core Systems Resource Management. In: ASPLOS (2018) 16. Salloum, C.E., et al.: The ACROSS MPSoC – A New Generation of Multi-core Proces- sors Designed for Safety-Critical Embedded Systems. In: 15th Euromicro Conference on Digital System Design (2012) 13 Self-Aware MPSoC Architecture Extensions for Automotive Applications 183

17. Sarma, S., et al.: On-chip self-awareness using cyberphysical-systems-on-chip (cpsoc). In: CODES+ISSS (2014) 18. Sarma, S.: Cyber-Physical-System-On-Chip (CPSoC): An Exemplar Self-Aware SoC and Smart Computing Platform. Ph.D. thesis, University of California Irvine (2016) 19. Tennenhouse, D.: Proactive computing. Commun. ACM 43(5), 43–50 (May 2000), http: //doi.acm.org/10.1145/332833.332837 20. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 21. Zeppenfeld, J., et al.: Learning classiﬁer tables for autonomic systems on chip. vol. 2, pp. 771–778 (01 2008)

Tuning the Computational Effort: An Adaptive Accuracy-aware Approach Across System Layers

Michael Bromberger

Chair of Computer Architecture and Parallel Processing, Karlsruhe Institute of Technology, Karlsruhe, Germany [email protected]

Abstract. Accuracy-aware computing is a novel way to tune systems towards design goals. However, a system designer faces several challenges when applying accuracy-aware computing. Therefore, this paper presents an innovative methodology integrating accuracy-aware computing into systems and applications. Additionally, it proposes a solution to tune integrated accuracy-aware methods within a system according to inputs, constraints, and system states. This solution represents a self-adapting mechanism for accuracy-aware computing.

Keywords: Accuracy-aware Computing, System Tuning, Self-adaptation

14.1 Introduction

Traditionally, designers tune computer systems by finding a trade-off between con- flicting design goals, such as performance and energy consumption, at design time. Since it is not always possible to determine all possible situations and dynamics during design time, paradigms such as autonomic [9] and organic computing [22, 15] were invented. By using self-x properties, where x represents adaptation, configuration, optimisation, etc., these paradigms allow the system to react and optimise the design-time unknowns. However, independent of the point of time, design or runtime, when a compromise is made and despite the advances in technologies, algorithms, and design techniques, there is still a high need for more energy-efficient and high-performance systems. Therefore, a new promising methodology called accuracy-aware computing gets high attention in that field. Accuracy-aware computing determines a compromise of design goals while considering quality of internal and external results and hence this paradigm allows for accuracy/performance trade-offs. As part of accuracy-aware computing, approximate computing comprises methods and tuning strategies to trade off accuracy for performance or energy [5, 14, 23]. For approximation-tolerant applications, we can signif- 186 M. Bromberger

Fig. 14.1: A feature-based classification pipeline. icantly improve traditional design values using this novel methodology while having a small loss of the quality of result (QoR) [8, 2]. Let us consider an example of a feature-based classification pipeline, see Figure 14.1. The pipeline classifies an object within a binary image. It calculates features of the object and uses a support vector machine (SVM) to determine a class label. The computational effort of certain features (green) depends on the input. For a firm deadline, integrating accuracy-aware computing allows for tuning tasks within the pipeline to satisfy constraints. This probably leads to a QoR loss for the classification, but the system’s quality of service is still higher compared to missing the deadline. However, a system designer faces several problems when using accuracy-aware computing: • Where can he or she integrate accuracy-aware methods? • Which methods are beneficial for his or her application? • How can he or she tune these methods according to certain situations?

14.1.1 Related Work and Challenges

For controlling streaming applications, we can use calibration phases [19] or control theory [6] during runtime. A proactive control approach leads to a dynamical adaptation of existing application parameters that have an impact on the QoR [21]. The aforementioned approaches does not consider the current input, which is also important for our non-streaming application example. Most input-aware control approaches make a binary decision for using approximation for a task [10]. However, there is often some margin for an approximation rather than full or no approximation. For instance, a certain degree of approximation may be acceptable for calculating an object feature without an impact on the classification accuracy. It is important that this adaption is applied during runtime, since the suitable approximation depends on the input itself. On the hardware level, a pipeline of neural accelerators provides such an input-aware solution, which applies a fine-grained adaption of the approximation degree [4]. However, this domain-specific approach does not present an input-aware solution for other domains, such as our example. Laurenzano et al. [11] present the only complete input-aware and dynamical control approach for setting knobs at software level. They use a calibration phase per input, which introduces overhead, and do not consider knobs at other layers of the computing stack. Instead of controlling the QoR, approaches control the energy consumption [7] or 14 An Adaptive Accuracy-aware Approach Across System Layers 187 guarantee a hard deadline while optimising the energy consumption [3]. These approaches are not input-aware. In total, there exists no general approach that considers all important aspects together.

14.1.2 Contributions

This work proposes a novel methodology to help system designers integrate accuracy- aware methods into their systems. The methodology includes the design of novel accuracy-aware methods that increase the applicability scope of accuracy-aware computing and are usefully applicable to novel applications domains for accuracy- aware computing. Furthermore, this thesis proposes a novel tuning approach that ﬁnds valuable settings of all tuning parameters considering the current inputs, constraints, and system states during runtime and hence this approach adds a self- adaption capability to the system. It designs the tuning approach in a way that it has a vertical view on the integrated accuracy-aware methods. The vertical view considers a combination of methods on different system layers, such as software, architecture, and hardware layer. A combination of accuracy-aware methods for a task leads to a better overall solution. Besides the accuracy-aware tuning knobs, the presented approach can also tune conventional methods, such as dynamic voltage and frequency scaling (DFVS) and the usage of several cores. Additionally, it accounts for multiple objectives (execution time, energy consumption, and QoR). The main contributions of this work are: •Anadaptive and dynamical accuracy-aware tuning approach across system layers. This approach dynamically tunes accuracy-aware knobs (parameters) according to given constraints, system states, and inputs. Hence, it introduces self- adaptation into an accuracy-aware system (Section 14.2). • The design, implementation, and evaluation of novel accuracy-aware methods on different system layers. The proposed dynamical tuning approach tunes the available knobs of the accuracy-aware and conventional methods and thus increase the applicability scope of accuracy-aware computing (Section 14.3). • The evaluation of the proposed dynamical tuning approach and accuracy-aware methods (Section 14.4).

14.2 A Novel Methodology for Accuracy-aware Systems

This section brieﬂy presents the novel methodology realising an accuracy-aware system (see also Figure 14.2). The starting point is an approximation-tolerant application composed of tasks (Step A). A designer collects representative input data for the system (Step B). As this methodology assumes that there already exist an application, we can use this application to gather the representative inputs or monitoring data from the production use. Also, we can check the assumptions made during the 188 M. Bromberger

Fig. 14.2: Novel methodology to realise an accuracy-aware system. design-time step within a calibration phase, where we also execute the exact application to see how our approximations behave. Moreover, when there exist available resources, we can use them from time to time to verify the QoR and may re-build the extracted global configurations. For instance, this is the case, when we must meet a deadline, but can continue the execution without an impact on the system performance. The next step for a designer is integrating accuracy-aware methods per suitable task (Step C). An important property of a suitable task is its significant contribution to the entire execution time or to the energy consumption. Steps A to C require the most effort for a designer. However, the corresponding thesis includes best practices for the proposed novel accuracy-aware methods to reduce the design effort. The remaining steps involve identifying, designing, and implementing solutions, methods, and algorithms that help the designers. In the design-time step, the task flows within the application are determined (Step D). Task flows depend on the application parameters that, for instance, control the ordering of image filters. To model the different task flows, an approach based on a decision tree classifier [16, 13] is used. Step F requires this information to extract the different task flows that use the approximate tasks and Step G determines a global performance profile for the complete task flow. Additionally, the runtime control mechanism (Step J) uses the information about the tasks flows to select one of the global performance profiles accordingly. Step E involves determining (near-)Pareto-optimal local configurations per task (a merged performance profile). Here, a local configuration represents knob settings for all integrated accuracy-aware and conventional methods. A Pareto-optimal local configuration means that this configuration can only be outperformed in terms of a single design value (either QoR or required budget). As a budget metric, my approach uses execution time, energy consumption, or a combination of both using energy delay product metrics. This local approach significantly reduces the number 14 An Adaptive Accuracy-aware Approach Across System Layers 189 of global configurations that must be considered for the entire application. To significantly further reduce the number of evaluated global configurations to determine a (near-)Pareto-optimal global configuration front, the corresponding thesis proposes a greedy-based static tuning algorithm. This algorithm itself determines a difference quotient of the performance profile for each task and current budget on demand. The rationale is that the actual performance profile depends on the approximation degree of the task’s input and Step E only assumes that the inputs are correct. The application input is assumed to be always exact. Calculating a conditional performance profile on demand would require a performance profile for each approximation degree. However, the merged performance profiles determined in Step E are used to extract a local configuration for the approximate task based on the granted budget. The extracted global configurations are selectable during runtime. Please remember that Step G is applied for each determined task flow in Step F. The proposed accuracy-aware approach introduces two different techniques to integrate input awareness into the system. The first creates a model that determines the required budget to reach a pre-defined QoR for a certain input and task (Step H). The model uses the QoR and input-dependent parameters of the function in question to determine the budget. Input-dependent parameters are the size of the input, the mean, the variance, or other statistical features that can be derived from the function’s input data. The proposed approach uses M5 models [17] to generate models. M5 models are tree-based models that divide the input space into subspaces and use a linear regression for each subspace. M5 models can be generated automatically [18]. How- ever, a system designer can easily decide to use other machine learning algorithms instead of M5 models. The second method is a monitoring approach that calculates features during the execution (such as the amount of considered pixels or the variation to the previous result) and uses these features to estimate the current QoR of a task (Step I). If the desired QoR is reached, the execution is terminated, and thus the monitor acts as a termination condition of the task. During runtime, a controller receives information about the constraints and system state (Step J), which it uses to select a global configuration. Input awareness is introduced on a local per-task level, and two different control modes exist: QoR mode and budget mode. In the QoR mode, each task reaches its specified local QoR even when requiring greater spending than initially provided. This decision depends on the input-aware local models (acquired in Step H). The information of the local desired QoRs is also part of the global configuration. For the budget mode, the user or a high-level controller grants a global budget (for instance, a deadline that has to be met) and thus an initial global configuration is chosen with the highest QoR on average for the given global budget. If several inputs processed by different application instances share a budget, unused local budgets is redistributed to other tasks. Unused budgets occur when the model determines a lower budget than that granted to the task initially. Finding a universal configuration for independent application instances or accuracy-aware applications is a multiple-choice knapsack problem (MCKP). The solution parameterised according to the current system state and constraints, leads to suitable global configurations for all instances when solving at runtime. 190 M. Bromberger 14.3 Innovative Accuracy-aware methods

To increase the applicability scope of accuracy-aware computing, the thesis designs novel methods for the corresponding thesis. This section briefly describe these methods and their respective application domain in the following. On the hardware layer, the thesis considers converting data items before transferring it to memory. It applies this method to algorithms working with floating-point values (for instance, image processing). Converting data items before the transfer to a representation with lower bits, leads to energy savings and an overhead reduction for data transfers while providing a higher accuracy compared to a native execution on a smaller data type [1]. The reason is that we perform internal operations using higher precision. On the architecture layer, it proposes a set of novel approximate computing-based design patterns for hardware accelerators. These patterns target dynamic programming algorithms and lead to more efficient designs. A combination of this approach and approximate data compression further improves the performance. On the software layer, it is essential to have a generally applicable software-only accuracy-aware method. Contract algorithms are employed in a new way to enable a clear, fine-grained, and predictable correlation between required budget and desired QoR. Tasks from different domains can be reimplemented as contract algorithm and thus the method represents the desired general software method. These innovative contract-based tasks pose the method used in the evaluation part.

14.4 Evaluation and Results

Using the proposed methodology in Section 14.2 and the methods in Section 14.3, the feature-based pipeline is transformed (see Figure 14.1) into an accuracy-aware system. The proposed greedy-based algorithm was used to extract suitable global configurations during design time. For the experiments, I used varying amounts of inputs and a different deadline per test run to evaluate how the system adapts to different system states. The input data was gathered during a productive use of a system. We see in Figure 14.3a that the conventional pipeline (gray, dashed line) is not able to meet the deadline (black, dotted line) in most situations and thus no classification results are available for these situations. In terms of execution time, the proposed monitoring method together with the realised accuracy-aware system (blue, solid line) is able to efficiently tune the system to satisfy deadlines. Even in cases where the conventional application meets all deadlines, we achieve an acceleration of about 4× with a negligible QoR loss (cf., for instance, test run 0 in Figure 14.3b). In extreme cases, the monitoring approach has a quality loss of 6%. In terms of QoR, the budget mode performs better but introduces a higher overhead (cf. Figure 14.4). This higher control overhead may require to select a configuration requiring less execution time to account for very tight deadlines while providing a lower classification accuracy (cf. test run 14 and 16). 14 An Adaptive Accuracy-aware Approach Across System Layers 191

Conventional Deadline Conventional Deadline Monitoring Monitoring 1.0

150 0.8

0.6 100 0.4 50

Execution Time [ms] . Classiﬁcation Accuracy 0 2

0 10 20 30 0 10 20 30 Test run id Test run id (a) Execution time. (b) Classiﬁcation accuracy.

Fig. 14.3: Conventional compared to accuracy-aware application.

Monitoring Budget mode Monitoring Budget mode 50

40 0.98

30 0.96 20 Execution Time [ms]

Classiﬁcation Accuracy 0.94 10 0 10 20 30 0 10 20 30 Test run id Test run id (a) Execution time. (b) Classiﬁcation accuracy.

Fig. 14.4: Comparison between the monitoring approach and budget mode.

The hull task (that is one of the first steps within the feature-based classification pipeline, see Figure 14.1) has the greatest computational effort and thus limits the performance for the accuracy-aware application. Therefore, a vertical view is applied to the tuning process. Accuracy-aware methods were integrated at different system layers into the hull task. Loop perforation (LP) [20] and approximate DRAM [12] are suitable methods to combine with the contract-based method. LP skips iterations within a loop specified by a perforation rate to trade off QoR of the loop for performance. Hence, we can use LP to reduce the computational effort of loops within a contract-based task. Approximate DRAM can reduce the required energy of the memories within the system, for instance, by increasing the refresh cycle time, which 192 M. Bromberger

Monitoring Deadline Monitoring Vertical view Vertical view 50 1.00

40 0.98 30

20 0.96

Execution Time10 [ms]

Classiﬁcation Accuracy 0.94 0 0 10 20 30 0 10 20 30 Test run id Test run id (a) Execution time. (b) Classiﬁcation accuracy.

Fig. 14.5: A vertical tuning view. may lead to data corruption. Hence, it represents an orthogonal method but may impact the performance profile of a contract-based task. Exploiting approximate DRAM and loop perforation, my approach extracts better global configurations using the greedy-based tuning algorithm. In Figure 14.5, we see that the vertical view (red, solid line) significantly reduces the execution time compared to the monitoring method (blue, dashed line). For tight constraints (run 10 to 20), it seems that the vertical view requires more time. However, this view avoids an unnecessary QoR degradation for these situations. Instead of 4%, we only have a loss of ≤ 1%.

14.5 Conclusion

This work presents a novel methodology realising accuracy-aware systems. The proposed methodology helps designers integrate accuracy-aware methods into their applications. The article introduced a dynamical tuning approach that is useful for designers to find suitable settings for the knobs of the integrated accuracy-aware and conventional methods. This approach considers the system state, multiple constraints, and inputs during runtime to improve the unconstrained design goals. Ex- ploiting this approach for a feature-based pipeline, the computation is accelerated by a factor of 4 without introducing a significant QoR loss. The dynamical tuning approach satisfied the deadlines while only having a small QoR loss. Moreover, a vertical view on the tuning process leads to a better solution. A limitation of the approach is that extracted configurations depend on the current used computing system. Changing the underlying computing system may lead to a 14 An Adaptive Accuracy-aware Approach Across System Layers 193 re-build of the input-aware units and the configurations itself. The corresponding thesis already proposes methods to reduce the overhead for the rebuild. However, future work must research for ways to increase the portability. Additionally, a weakness is the required programming effort for programmers and designers. While the thesis already presents automatic solutions and best practices, the integration of the methods is done by the designer. Therefore, future work shall design tools that support designers.

References

1. Bromberger, M., Heuveline, V., Karl, W.: Reducing energy consumption of data transfers using runtime data type conversion. In: International Conference on Architecture of Computing Systems (ARCS). pp. 239–250 (2016) 2. Chippa, V., Chakradhar, S., Roy, K., Raghunathan, A.: Analysis and characterization of inherent application resilience for approximate computing. In: 50th ACM/EDAC/IEEE Design Automation Conference (DAC). pp. 1–9 (5 2013) 3. Farrell, A., Hoffmann, H.: Meantime: Achieving both minimal energy and timeliness with approximate computing. In: 2016 USENIX Annual Technical Conference (USENIX ATC 16). pp. 421–435. USENIX Association, Denver, CO (2016), https: //www.usenix.org/conference/atc16/technical-sessions/presentation/farrell 4. Grigorian, B., Farahpour, N., Reinman, G.: Brainiac: Bringing reliable accuracy into neurally-implemented approximate computing. In: 2015 IEEE 21st International Sym- posium on High Performance Computer Architecture. pp. 615–626 (Feb 2015) 5. Han, J., Orshansky, M.: Approximate computing: An emerging paradigm for energy- efﬁcient design. In: 2013 18th IEEE European Test Symposium (ETS). pp. 1–6 (5 2013) 6. Hoffmann, H.: Coadapt: Predictable behavior for accuracy-aware applications running on power-aware systems. In: 26th Euromicro Conference on Real-Time Systems. pp. 223–232 (7 2014) 7. Hoffmann, H.: Jouleguard: Energy guarantees for approximate applications. In: Proceed- ings of the 25th Symposium on Operating Systems Principles. pp. 198–214. SOSP ’15, New York, NY, USA (2015) 8. Hoffmann, H., Sidiroglou, S., Carbin, M., Misailovic, S., Agarwal, A., Rinard, M.: Dy- namic knobs for responsive power-aware computing. In: Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems. pp. 199–212. ASPLOS XVI, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/1950365.1950390 9. Kephart, J.O., Chess, D.M.: The vision of autonomic computing. Computer 36(1), 41–50 (Jan 2003) 10. Khudia, D.S., Zamirai, B., Samadi, M., Mahlke, S.: Quality control for approximate accelerators by error prediction. IEEE Design Test 33(1), 43–50 (2 2016) 11. Laurenzano, M.A., Hill, P., Samadi, M., Mahlke, S., Mars, J., Tang, L.: Input responsiveness: using canary inputs to dynamically steer approximation. ACM SIGPLAN Notices 51(6), 161–176 (2016) 12. Liu, S., Pattabiraman, K., Moscibroda, T., Zorn, B.G.: Flikker: Saving dram refresh- power through critical data partitioning. In: Proceedings of the Sixteenth International 194 M. Bromberger

Conference on Architectural Support for Programming Languages and Operating Sys- tems. pp. 213–224. ASPLOS XVI, ACM, New York, NY, USA (2011), http://doi.acm. org/10.1145/1950365.1950391 13. Mitra, S., Gupta, M.K., Misailovic, S., Bagchi, S.: Phase-aware optimization in approximate computing. In: IEEE/ACM International Symposium on Code Generation and Optimization (CGO). pp. 185–196. IEEE (2017) 14. Mittal, S.: A survey of techniques for approximate computing. ACM Computing Surveys (CSUR) 48(4), 62 (2016) 15. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 16. Quinlan, J.R.: Induction of decision trees. Machine learning 1(1), 81–106 (1986) 17. Quinlan, J.R., et al.: Learning with continuous classes. In: 5th Australian joint conference on artiﬁcial intelligence. vol. 92, pp. 343–348. Singapore (1992) 18. Research, R.: Cubist. https://www.rulequest.com/cubist-info.html (2016) 19. Samadi, M., Lee, J., Jamshidi, D.A., Hormati, A., Mahlke, S.: Sage: Self-tuning approximation for graphics engines. In: Proc. 46th Annual IEEE/ACM International Symposium on Microarchitecture. pp. 13–24. ACM (2013) 20. Sidiroglou-Douskos, S., Misailovic, S., Hoffmann, H., Rinard, M.: Managing performance vs. accuracy trade-offs with loop perforation. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. pp. 124–134. ACM (2011) 21. Sui, X., Lenharth, A., Fussell, D.S., Pingali, K.: Proactive control of approximate pro- grams. SIGOPS Operating Systems Review 50(2), 607–621 (Mar 2016) 22. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 23. Xu, Q., Kim, N.S., Mytkowicz, T.: Approximate computing: A survey. IEEE Design Test 33(1), 8–22 (Feb 2016) Part V

Organic Computing in Real-world Applications

Dynamical Evacuation Route System

Tim Wachter¨

Institute of the Intelligent Building, University of Applied Science Bielefeld, Campus Minden, Minden, Germany [email protected]

Abstract. This work examines different approaches in the area of dynamical evacuation systems in smart buildings from an Organic Computing perspective. The focus lies on how self- organised evacuation route systems handle structural building information and which kind of evacuation path planning algorithm they use. Another important aspect is which kind of information are typical for a self-optimising dynamical evacuation route system. Subsequently, the developed prototype is explained and evaluated that made it possible to handle the structural conditions of a building in a mathematical graph in order to calculate safer escape routes. Finally, different approaches are discussed that will be investigated in the future.

Keywords: Evacuation Route System, Emergency Management, Organic Computing

15.1 Introduction

The global population is growing steadily, and the living space is becoming ever scarcer. Buildings are getting higher and more complex, that includes that the number of people living in a single place increases steadily. Therefore, it becomes more important to guarantee an efficient, safe and reliable building evacuation system. Organic Computing (OC) are characterised by self-x properties such like self- administration, self-healing and self-optimisation [8, 11]. All these properties should be guaranteed in a dynamic escape route system so that the system can automatically react to all eventualities in an emergency situation. The state of the art in the field of building technology makes it possible to use environmental information inside the building to calculate dynamical evacuation routes [13]. Today, the evacuation routes are static and offer just the shortest way out of the building [5]. In the worst case, the persons to be evacuated are led directly to the source of the fire or through a blocked corridor. This means that temporary conditions such as the capacity reduction of the floor by cartons of paper, robots or meal trolleys are not considered in the escape routes. Another disadvantage of the widespread fire control is that every person en- tering a building needs to check the escape route plan actively and memorise the 198 T. Wachter¨ shortest route out of the building. Another issue arises, when an emergency situation occurs. Occupants might be in panic, might not remember the right evacuation path and might turning in the wrong direction. As a further problem, the escape signs are usually mounted on the ceiling and are therefore barely visible in dense smoke. Thus, we propose to calculate the evacuation routes dynamically and make use of alterable signages under consideration of environment information such as smart sensor and actors. In addition, we propose to place the signages on the floor of the building, as it can also be found in airplanes.

15.2 Related Work

Emergency Management with a focus on building evacuation is an emerging research topic in computer science. The focus of the relevant literature is mostly on the evacuation path planning algorithm and the data structure which means how to store building information. In the preliminary work, we ascertain, that in the most cases the focus is on the calculating of the shortest way out of the building. The Dijkstra and A* algorithms are often used as a basis and adapted with relevant parameters such as the length or capacity of the corridor [7, 10, 12]. In most of the examined work, mathematical graphs are used for the representation of corridors, rooms or crossroads [4, 10, 12]. Weifang and Qiang [12] developed an intelligent fire evacuation system which is based on a weighted graph. The graph has the following three different kinds of vertexes: source vertex, transportation vertex and target vertex. A source vertex is the starting point for an evacuation path and has only outcoming edges such as an office. At the end of an evacuation path is a target vertex. This could be a staircase or an exit and has only incoming edges [12]. Guo, Gao, Ni et al. [4] developed a broadly similar mathematical graph such as Weifang and Qiang [12]. An important difference is the edge weight which is based on the corridor capacity and the time which an evacuee needs to pass the corridor. Another approach is to use two static graphs instead of one. The first graph is the navigation graph and represents the building layout which includes a set of sensors and the paths in the building where people can walk. The other graph is called hazard graph and has the same vertexes as the first graph. The difference between the navigation and the hazard graph is the calculation of the edge weight. The hazard edge weight is the time which the fire needs to spread from one connected sensor to another and the navigation weight is the time which an evacuee needs to move from one corridor to another [4]. The approaches mentioned so far are based on mathematical graphs and have been developed for building components such as smart lights or signs. Wu, Yu, Chine et al. [13] are pursuing an approach where the evacuation route is displayed on a smart handheld device. This system is based on four different modules. The first module is the location module which analyses the location of the smart handheld device inside the building via radiofrequency identification (RFID). The environment sensing module collects the building information from the different sensors such as humidity, 15 Dynamical Evacuation Route System 199 luminance and temperature via ZigBee wireless sensor network. The central control module integrates all modules in the system and calculates the evacuation paths. The last module is the mobile application which displays the evacuation path direction as an arrow. The arrow has three different states. Red if the user is near to the fire. Green if the user is near to the exit and yellow if the user is between the fire and the exit [13]. The disadvantage of this system is that the people inside the building have to hold their smartphones and they have to concentrate on the smartphone and the environment at the same time. So far, only systems that are based on a central server architecture have been discussed. Chen, Cheng and Tseng [2] took another approach and developed a distributed emergency guiding prototype for building evacuation. The focus of the work is on optimising evacuation time of the building and is based like many mentioned works before on a wireless sensor network. Each sensor has a parameter which describes the degree of danger. Sensors near to the exit have smaller parameter values, whereas sensors in the near of the hazard have a higher parameter value. For the building representation a graph with undirected edges in two dimensions is used. The graph supports two different kinds of nodes: exit nodes and normal way nodes. Each exit node creates an emergency guiding tree by itself. For the edge weight calculation, the parameters corridor capacity, corridor length, exit capacity, concurrent move and people distribution are used. The occupancy detection is realised via RFID and image recognition. The system calculates the escape route when an emergency event is thrown. If the hazard area is spreading, the algorithm is rerun. Another approach aims to evacuate a building in the shortest time and calculate the distribution of evacuees based on the capacity of the corridor. Furthermore, moving speed, distance to the exits and capacity of the exits are considered for the calculation. The information is organised on the basis of a grid topology [3]. The disadvantage of the frameworks is that only 2D indoor environments can be modelled [2, 3]. Therefore, both systems are not able to calculate escape routes for multi-storey buildings. For this challenge Bai, Huo, Zou et al. [1] used FDS+EVAC which is a simulation software in the field of fire safety. Additionally, the system simulates the flow of people in the building. Another simulation tool for measuring the influences during a fire situation is the Consolidate Fire and Smoke Transport (CFAST) fire zone model[6]. Kulakowski, Andrzej, Rogala et al. [7] developed likewise a simulator which doesn’t focus on creating corridors in the form of edges, instead, they represent the surface of the building in which agents (people inside the building), obstacles and emergency exits are placed. If a floor has been recreated in the simulator, an external programme automatically creates the missing connection between the obstacles and creates a new triangular grid for the floor. All of these three approaches aim to be a simulation for fire scenarios but not a complete escape route system for a building that might be used in the future. Therefore, it will be difficult and complicated to validate the findings from the simulation software in the real world. 200 T. Wachter¨ 15.3 Approach

The unique feature of the simulator compared to the approaches discussed in Sec- tion 15.2 is that it allows considering the different building characteristics (lighting, accessibility, fire compartments, etc.) in the escape route computation. The approach is that the system reacts dynamically to the spread of the fire and includes the structural conditions of the building in the calculation. When an emergency occurs, the simulator localises the fire source and calculates automatically a safe evacuation path to an emergency exit. The dynamic evacuation system calculates optimal evacuation routes to safe locations under consideration of the basic technical rules of the Federal Institute for Oc- cupational Safety and Health1. These technical rules fall into various categories such as barrier-free design of workplaces, rules for windows, skylights and translucent walls, sanitary facilities, escape route, emergency exit, rescue and evacuation plan [5]. For an efficient building evacuation, it is important to comply with the conditions of the technical rules from the categories escape route, emergency exit, rescue and evacuation plan. An essential rule for our developed algorithm is that the escape route leads to a safe area which means outside of the building or an area where people are temporarily safe. Another important rule is that the length of the escape route is not exceeded. The permitted length of an escape route is defined as the beeline from whereabouts to an emergency exit. Table 15.1 lists the provisions of the maximum escape route length of the different categorised rooms. We assume that the current simulator represents an office building which means all rooms have a normal risk of fire. So it is not possible to create several types of rooms in the simulator. By German law, an escape route length for normal risk rooms is allowed up to 35 meters. Hence in the future, the simulator should be able to manage different types of rooms with different maximum escape route lengths.

Table 15.1: Provisions for the escape route length [5]

Characteristic Length a) Rooms without or with normal fire risk, except rooms according Up to 35m to b) to f) b) Fire-prone rooms with automatic fire-extinguishing system Up to 35m c) Fire-prone rooms without automatic fire-extinguishing system Up to 25m d) Rooms which are toxicant endangered Up to 20m e) Explosion-prone rooms, except rooms according to f) Up to 20m f) Rooms containing explosive substances Up to 10m

1 https://www.baua.de/DE/Angebote/Rechtstexte-und-Technische-Regeln/Regelwerk/ASR/ ASR-A2-3.html (last access: 09/09/2018) 15 Dynamical Evacuation Route System 201

The developed system is based on a mathematical Graph G =(V,E) with corridor crossings (V) and corridors (E). Each edge (corridor) has the attributes length, evaluation, start − position and end − position. The attribute length contains the length of the corridor (E). A description of the quality of the corridor is stored in the attribute evaluation which will be explained later. The values start − position and end − position are representing the connection points and the position of the edge which is used for the determination of the path direction. A vertex contains a list in which the affected corridors are registered. This metadata is relevant for structuring and visualisation of the graph and the connection of different ﬂoors. To determine the evaluation attribute, a questionnaire was developed in cooperation with civil engineers. It contains 56 parameters in the three main categories corridors, staircases and utilisation units. This will enable us to evaluate the different elements in a building separately and individually in the future. An excerpt from category corridors is given in the following table.

Table 15.2: Excerpt from the category corridors as part of escape routes in the building.

General criteria Yes No Does the corridor allow to escape in two directions? 3 1 Is the corridor a necessary corridor according to building regulations? 3 1 Leads to a necessary staircase or outside the building? 3 1 The fire resistance class of the corridor partitions corresponds to the 31 requirements of the building supervision? Accessibility of the corridor Is the corridor clear and well lit? 3 1 Is there a sequence of less than 3 stairs (trip hazard) in the corridor? 1 3 Corridor lighting Does the corridor have natural lighting? 2 1 Does the corridor have electric lighting? 2 1 Does the corridor have emergency lighting? 3 1 Smoke-free staircase, formation of smoke and fire compartments Are there any flammable objects in the corridor? 1 3 Are the ends of the corridors regularly closed (windows and doors)? 0 3 Is there a division into smoke compartments with permissible section 31 lengths? Do the types of corridor end to the staircases and other rooms corre- 31 spond to the requirements?

Each of these parameters (formulated as questions) is answered with yes or no and gets a weighting in the interval [0,3]. The maximum value 3 is considered to be the 202 T. Wachter¨ best, whereas a value of 0 is interpreted as the worst. Questions that are less relevant can have a maximum value of 2. After all questions have been answered, a total value can be determined. The ﬁrst idea was to multiply and normalise all values. The problem was that this did not include the weight of the individual questions. For this reason, the maximum for each question has been included in the formula to have a percentage value for each question (n: number of questions, x: answer value, y: maximum answer value). The product is calculated and standardised overall percentage values. The value range for the evaluation value is between 0 and 1. The higher the value, the more suitable ist the corridor as an escape route.

n x ∏ i (15.1) i=1 yi

The calculation of the dynamic escape routes is based on the Dijkstra algorithm which has been slightly modified because there can be more than one emergency exit in a building. For this purpose, all routes to the emergency exits are calculated and compared and the best one is used. For the best route, the algorithm compares the evaluation values first and then the length of the corridor. Several test series were carried out to validate the algorithm. The dynamic escape route was compared with the static one (the shortest route out of the building or into a safe area). The parameter evaluation was tested with values of 0,0.3,0.5 and 0.7. In this case, a total of 292 ∗ 4 = 3364 results are available because the floor contains 29 guiding lights (corridors) and were tested with four different values. The escape routes were calculated on the basis of the floor plan depicted in figure 15.1. In this figure, the emergency exits are indicated by the number one and the start position for the calculation by two. In figure 15.2 the best way is blocked by the fire and in figure 15.3 the escape routes from figures 15.1 and 15.2 are also not accessible. Many results are not relevant for the comparison of dynamic and static escape routes, as the fire position often does not interfere with the escape route. Because, in most cases, the dynamic and static escape routes are identical. However, the advantage of the calculated escape routes is that these never guide the persons through or to fire. The only exception is when an ethical dilemma arises which is explained at the end of the chapter. Figure 15.3 demonstrates that if a corridor has a worse evaluation value (the corridor with the orange lights), it is not chosen as an escape route even though the corridor has a shorter length. In the experiment, we have examined all possible combinations of a disadvantaged corridor and not disadvantaged ones. The results shows if only one corridor has a poor evaluation value, this corridor will not be used for the evacuation path. Therefore, it is sufficient to consider the results on the basis of the evaluation value 0.5. Figure 15.4 shows an overview of the lengths of the calculated escape routes. In the test series, each possible combination of a burning and a disadvantaged corridor was simulated, resulting in a number of 841 escape routes. The X-axis represents the number of escape routes and the Y-axis shows the route length. The red line indicates the maximum escape route length of 35 meters. The figure shows that 651 15 Dynamical Evacuation Route System 203

Fig. 15.1: The basic start sce- Fig. 15.2: The fastest path Fig. 15.3: The best and sec- nario/floor blocked ond best path blocked of all escape routes correspond to the optimum (77.4%) which in this case is the shortest escape route not affected by fire. There are 94 alternative routes below the maximum length and a total of 65 escape routes above the maximum (36 to 37.5 meters). It is noticeable that only two escape routes distinctly exceed the maximum length. In 29 cases no escape route was calculated because the evacuees were in a dead end and the escape route would lead directly through the fire. So the algorithm would have to make an ethical decision. In this case, an ethical decision means that the evacuation system must decide if it is more appropriate to guide through the fire or guide as far away from the fire as possible, for example in the corner of the room. In such a decision, various factors have an impact, like the size of the fire, the speed at which the fire propagates, the oxygen concentration in the ambient air, the possibility of rescuing the person from outside the building (through windows), the time it will take for the fire brigade to rescue the person, and so on.

Fig. 15.4: Number of calculated escape routes in relation to the length 204 T. Wachter¨ 15.4 Future Work

This section explains the next steps and subgoals for the overall project. On the one hand, the next steps for the prototype mentioned in the section before are described. On the other hand, further approaches are explained how the system can be tested in the future under real conditions.

15.4.1 Prototype extension

In the above mentioned prototype, there are two optimisation points which are shown in Figure 15.5 and 15.6. Each edge E =(len,eval) in the graph has a length (len) and an evaluation value (eval). The first challenge arises when several parameters are used for dynamic escape route calculation to set them in an optimal ratio. Because the algorithm has to decide which is more important, either the length of the escape route or the evaluation value and in which case it makes sense to choose a path that has a slightly worse evaluation value but is much shorter instead of the longer path with a better evaluation value. In Figure 15.5, an escape route from node S to node E will be calculated. Our prototype considers the length first when calculating the escape route and if there are several routes of the same length, the evaluation value will be compared. The algorithm selects the escape route via node B because the escape route via node B is shorter than via node A and there is no other route with the same length to compare the evaluation value. In this case, it would make more sense to lead the escape route via node A because this route has a much better evaluation value and is just 2 meters longer. The second optimisation option is demonstrated by the figure 15.6. In this case, the escape route from node S to the exit node E should be calculated. The algorithm decides for the green marked escape route which leads over the node B because in this case the length and the evaluation value is the same for the route for both nodes A and B. At this point, it would be more reasonable to use the escape route via the node A because this route is farther away from the source of the fire (node C)asvia node B which means the fire needs more time to interfere with the escape route. Guo, Gao, Ni et al. [4] developed a framework that only simulates escape routes for 2D environments because they only use two types of nodes in their graph. We solved this problem in our prototype by creating an additional node type. Figure 15.7 shows an undirected graph which represents a two-storey building. The grey nodes are the normal connection points of one or more corridors. The nodes with the green border color represent the emergency exits and the orange node is the new type of node and represents a staircase which connects the floors. In this case, the node is also an emergency exit. According to the fire protection regulations of ASR 2.3, a staircase is regarded as a safe area [5]. Thus the algorithm of the prototype calculates the escape routes just from the first floor to the staircase. If a staircase has no direct exit to the 15 Dynamical Evacuation Route System 205

Fig. 15.5: Challenge of the conditions of the Fig. 15.6: Same route length and evaluation individual parameters value basement, the algorithm would have to calculate the escape route further to the exit nodes in the basement which will have to be realised in the future.

Fig. 15.7: Support of multi-storey buildings

15.4.2 Distributed System

The next steps are developing the simulator as a distributed system and distributing the calculation of the escape routes to thee nodes of the graph in the system. This offers many advantages, for example the distribution of computational capacity, reliability, scalability that avoids the need for a powerful central server redundancy. The idea of shifting the calculations of escape routes to the individual nodes is intended to create a high-performance system that can also be used in complex buildings such as skyscrapers and airports. A further advantage is the expandability and scalability so that the effort for adding a new part of a building is low. In this case, it is reasonable to use Wireless Sensor Network [2, 3]. One of the most important advantages is reliability. If a central server fails, the complete evacuation system does 206 T. Wachter¨ not work, or if a component, i.e. a corridor, fails in the distributed system, the worst case is that the corridor is inaccessible and the rest of the system functions normally. The challenge here is to develop an efﬁcient algorithm that guarantees that all nodes know their neighbourhood relationships. This means that each node knows the world in which it is located. For ﬁrst experiments, the Bellman-Ford or the Distance-Vector algorithm could serve as a basis. However, the algorithm would need to be adapted because at least two parameters, the length of the edge and the description of the edge by the evaluation value are considered in the calculation of the evacuation paths.

15.4.3 Fire and smoke propagation simulation

In order to ensure the highest possible degree of safety during the evacuation of a building, it is necessary to consider not only the length of the corridor and the structural conditions of the corridor but also the spread of fire and smoke propagation. A simple and not close to reality fire propagation was realised by Kulakowski, Andrzej, Rogala et al. [7] in form of an ellipse which propagates depending on time. In the approach of Guo, Gao, Ni et al. [4] the fire propagation is observed in the simulator. For this purpose, the speed is measured which the fire needs to spread from one sensor to the next. The disadvantage is that only the fire and not the propagation of dangerous gases or the oxygen saturation in the ambient air is considered. The idea is to include both fire and smoke propagation in the calculation of safe escape routes. Because of the possibility to consider the structure conditions of the individual corridors, like ”Are there any flammable objects in the corridor?” or ”Is there a division into smoke compartments with permissible section lengths?”,we assume that a more accurate prediction of the spread of fire and smoke is possible. In the first step, as in the simulator for the propagation of methane in coal mines of Shangliang, Xinye, Mei et al. [9] the wind flow direction and wind speed will serve as basic information and will be combined with further information such as temperature, oxygen saturation in the ambient air, carbon monoxide, carbon dioxide and structural conditions. Furthermore, it will be investigated which parameters are needed for a more accurate prediction.

15.4.4 Ground Truth

One of the biggest challenge in this research area will be the evaluation of the made assumptions.The problem is that fire scenarios are difficult to simulate in a realistic environment and fire simulations are used as exercises for the fire brigade. Another problem is that the people inside the building should keep in mind the existing escape routes. A further challenge will be the fulfilment and compliance with the legal requirements applicable in Germany. The idea is that the system which was tested successfully in the simulator should first be used under laboratory conditions. If the system performs as expected, the next step would be to contact the fire brigade and 15 Dynamical Evacuation Route System 207 the University of Applied Sciences Bielefeld in order to create or use a possible environment in that the system can be tested under realistic conditions. This could be used for example in combination with the recurring fire drill on campus. In the current existing buildings, there are usually no smart products installed for escape route control, let alone networked hardware. For this reason, a small cube is to be developed which is equipped with sensors and actuators in order to collect environmental information and display escape routes without structural intervention.

15.5 Result and Conclusion

This paper presents the state of the art in the field of dynamic evacuation route calculation with an emphasis on building representation. We summarise which kind of evacuation path planning algorithms and environmental information are used in current approaches. Aforementioned work have all the same goal: To evacuate people from an emergency situation in the best possible way. Most systems use weighted graphs to represent a building. Rooms, exits and crossings are depicted as vertexes and corridors as edges. The advantage of this structure is that the edge weight can be a good indicator for the corridor quality but it is noticeable that in the researched literature the parameters for the edge weight calculation are different. Especially because the systems are based on various assumptions. For example Tabirca, Brown and Sreenan [10] disregard the corridor capacity at the edge weight calculation because they assume that the number of people is not bigger than the capacity of a corridor. Guo, Gao, Ni et al. [4] consider for the edge weight, the corridor capacity and the time which an evacuee needs to pass the corridor. Thus, it is essential to define a set of different environmental information from sensors (temperature, CO, CO2, presence detector, etc.) and building conditions (stairs, handrails, corridor capacity, etc.) to calculate a balanced edge weight. The capacity is often statically calculated and based on the corridor width. In an emergency situation, a sufficient evacuation path could be clogged because the algorithm ignores the characteristics of a floor or the temporally obstacles such as food trolleys, bikes, litter and so on. Most systems use a central server that calculates the escape routes for the entire building or smart handheld device and similar path planning algorithm. We have developed an evaluation matrix to describe the characteristics of a corridor which bases on the fire protection regulations of the German Federal Institute for Occupational Safety and Health. It was established that the dynamic escape routes do not differ from the static escape routes due to the consideration of the structural conditions, as the fire does not affect the escape route. However, in cases where the shortest escape route is blocked by fire, the escape route is only minimal longer and shorter than the maximum length of 35 meters. Additionally, it can be noted that it is important that the algorithm evaluates the characteristics of the corridors continuously and it is not sufficient to include only the static information at the system startup. It makes sense to consider temporary obstacles such as food trolleys or cardboard boxes in the calculation because these could 208 T. Wachter¨ reduce the capacity of a corridor and thus clog the corridor. Another important aspect for the future is, that it will be necessary to find a ratio between the dynamic evaluation value and the length of the corridor which will be used as an edge weighting. In addition, the existing prototype is to be converted into a distributed system in the future.

15.6 Acknowledgement

This work is supported by the Ministry of Culture and Science of the German State of North Rhine-Westphalia.

References

1. Bai, W., Huo, Y., Zou, G.W., Gao, Y.: Simulation of fire evacuation in a high-rise office building. In: IEEM 2015. pp. 1704–1708. IEEE, [Piscataway, NJ] (2015) 2. Chen, L.W., Cheng, J.H., Tseng, Y.C.: Evacuation time analysis and optimization for distributed emergency guiding based on wireless sensor networks. In: 2012 International Conference on Connected Vehicles and Expo (ICCVE). pp. 130–135. IEEE (2012) 3. Chen, P.Y., Kao, Z.F., Chen, W.T., Lin, C.H.: A distributed flow-based guiding protocol in wireless sensor networks. In: 2011 International Conference on Paralell Processing. pp. 105–114. IEEE Computer Society, Los Alamitos, Calif. (2011) 4. Guo, D., Gao, C., Ni, W., Hu, X.: Max-flow rate priority algorithm for evacuation route planning. In: 2016 IEEE First International Conference on Data Science in Cyberspace (DSC). pp. 275–283. IEEE (2016) 5. Health, Federal Institute for Occupational Safety and Health: Asr a2.3 escape routes and emergency exits, escape and rescue plan (2017) 6. Hu, Y., Wang, X., Wang, F.Y.: A quantitative study of factors influence on evacuation in building fire emergencies. IEEE Transactions on Computational Social Systems 5(2), 544–552 (2018) 7. Kulakowski, A., Rogala, B.: Agent simulation of the evacuation process from a building during a fire. In: 2017 12th International Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT). pp. 385–388. IEEE (2017) 8. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing - Technical Systems for Survival in the Real World. Autonomic Systems, Springer International Publishing, Cham (2017) 9. Shanliang, Y., Xinye, Z., Mei, Y., Yun, Z.: Interactive methane monitoring system modeling and simulation for coal mines. In: International Conference on Measurement, In- formation and Control (ICMIC), 2013. pp. 772–777. IEEE, Piscataway, NJ (2013) 10. Tabirca, T., Brown, K.N., Sreenan, C.J.: A dynamic model for fire emergency evacuation based on wireless sensor networks. In: 2009 Eighth International Symposium on Parallel and Distributed Computing. pp. 29–36. IEEE (2009) 11. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 15 Dynamical Evacuation Route System 209

12. Weifang, Z., Qiang, C.: Implementation of intelligent ﬁre evacuation route based on Internet of things. In: Xu, B. (ed.) Proceedings of 2015 IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC 2015). pp. 934– 938. IEEE Press, [Piscataway, New Jersey] (2015) 13. Wu, C.C., Yu, K.M., Chine, S.T., Cheng, S.T., Huang, Y.S., Lei, M.Y., Lin, J.H.: An intelligent active alert application on handheld devices for emergency evacuation guidance. In: The Fifth International Conference on Ubiquitous and Future Networks. pp. 7–11. IEEE, Piscataway, NJ (2013)

Robustness Modeling and Assessment of Interdependent Smart Grids

Ferdinand von Tullenburg¨

Salzburg Research Forschungsgesellschaft mbH, 5020 Salzburg, Austria [email protected]

Abstract. Future Smart Grids heavily rely on collaboration of two interdependent subsystems: the information and communication (ICT) system and the electrical system itself. The ICT system enables information flows between control centres and sensors in the field, while the grid provides necessary electrical power to the components of the ICT system. Past has shown that malfunctions in one of the systems can propagate into the other system leading to wide-spread and heavy outages of power supply. Within this PhD project, a methodology for modelling, assessing and quantifying the Smart Grids’ robustness under consideration of interdependencies between the ICT and electrical subsystem. A particular novelty to existing approaches is the application of machine-learning for robustness quantification based on the current state of both subsystems.

Keywords: Smart Grid, Robustness, Modelling, Quantiﬁcation, Interdependent Systems

16.1 Motivation

The information- and communication technology (ICT) and the electrical subsystem of modern power systems are interdependent. This means both subsystems have a direct or indirect influence on each other basically due to the fact that the ICT subsystem provides the necessary communication capabilities, while the electrical subsystem provides the necessary electrical power to ICT equipment. Consequently, they are considered to be “Interwoven Systems” [18, 3]. However, past has shown that those interdependencies may pave the way for failure cascades between both subsystems leading to uncontrolled behaviour and total loss of control [5]. For example, in September 2003, a ground fault of a transmission power line propagated into the ICT system causing power losses in central communication nodes. The consequence has been degraded functionality of the operators’ supervisory control and data acquisition (SCADA) system, which finally led to a complete loss of control. The failure cascade caused a large-scale blackout in Italy with about 56 million affected people [22]. Another example is the large-scale power outage in Northern America also 212 F. von Tullenburg¨ in 2003, where failures of the SCADA system led to severe contingencies. Soft- ware failures of the monitoring system resulted in degraded situational awareness preventing adequate grid control operation when several transmission lines where overloaded [1]. As seen, unpredictable events in one of the subsystems may affect the other subsystem with severe consequences on the stability of the power supply system. Thus, for grid operators, it is of considerable value to assess the robustness of power supply systems in case of random failures occurring in one subsystem propagating to the other. One particular challenge of handling these interdependencies is that they are subject to continuous change due to reconfiguration and alter- ing conditions. For instance, abrupt shifts of power consumption and production or topological changes due to component failures. Thus, in this work, a methodology is developed enabling to (1) identify interdependencies between the ICT and electrical subsystems during runtime under consideration of dynamically changing states and configurations, and (2) to assess the relevance of those interdependencies according to the overall system robustness. In view of the fact that influencing parameters such as system states and configurations vary widely, in this project (3) machine learning (ML) is utilised to provide instantaneous online estimations of system robustness. These ML-driven online estimations open perspectives even for automated reconfiguration for improving system robustness. For this, the online estimations could be integrated into an Organic Computing [21, 15] inspired system, e.g. following the design as proposed in [19], which has particularly proposed in the context of mutual influencing systems [20].

16.2 Research Questions

The project is subdivided into the following research questions: RQ0 - Top-level research question: How can robustness of power distribution systems be analysed during operation with machine-learning methods under consideration of interdependencies between electrical systems and ICT systems and current state and utilisation of both systems? In the context of electrical systems, robustness is usually defined as the capability of the electrical system to deliver electrical power supply within expected quality even in the presence of active failures or suddenly materialising incidents. The envisaged methodology aims at evaluating the robustness of the electrical system with a particular focus on the possible risk caused by interdependencies. In order to address the overall target, this PhD project is subdivided into three packages addressing the following research questions: RQ1 - Failure Classification: How is the overall distribution system robustness in- fluenced by incidents happening in the electrical subsystem or the ICT subsystem? For answering RQ1, a comprehensive classification of failure types and their effects on components of the smart grid is developed. The result of this part is an important 16 Robustness Modeling and Assessment of Interdependent Smart Grids 213 component for the development of the simulation model to examine cascading failure impacts. The result is a taxonomy and grouping of distinguishable failure types and scenarios. This is required later on for assessment the grids robustness against important failure scenarios. RQ2 - Interdependency Modelling: How to model interdependencies between the electrical and the ICT subsystem of a distribution system for providing a comprehensive and comprehensible representation of affected components and risks for system robustness? To answer RQ2 it is required to develop an approach for precisely modelling interdependencies between the ICT and electric subsystem by evaluating the dependencies between components of the respective subsystems. The envisaged model names and qualifies present interdependencies based on existing definitions (such as cyber, physical, geographical, and logical [16]) and is capable of expressing the quantified risk for system robustness. The result is a generic model of dependencies in smart distribution systems including a scheme for quantifying the importance or risk of dependencies. Pursuing that, a mathematical model is developed for determining the risk certain interdependencies have on system robustness. Various failure scenarios are considered that describe aspects such as component capacity/utilisation or the type and extent of dependencies of a component. The robustness quantification needs to be independent of a particular smart grid (size, type and distribution of included components) to guarantee generality and comparability. Later, this model is used for operational robustness assessment. RQ3 - Robustness Analysis: How can machine-learning help to instantaneously analyse and quantify the robustness of power supply systems considering current system state and configuration? Here, a method is developed to fill the interdependency model (RQ2) with values. For this, a co-simulation or integrated simulation for the electrical system is developed. The simulation comprises of models for relevant components including most important parameters. The simulation is used to evaluate different failure scenarios (an outcome of WP1 / RQ1) under different boundary conditions describing system utilisation and state. Using the mathematical model for interdependence description (RQ2), results on system robustness are evaluated. These simulation results are furthermore used for training a classifier. The classifier is finally supposed to quantify system robustness under use of real-time information on system state and configurations.

16.3 Related Work

Relevant related work is subject to the topics: failure classiﬁcation in power distribution systems, analysis of interdependencies in systems-of-systems, and quantiﬁcation of power grid robustness including the role of machine learning in that area. 214 F. von Tullenburg¨

A broad variety of failures can occur in the subsystems of smart grids. For the sake of analysing the robustness of a power system, it is therefore important to understand the causes and consequences of failures. This is particularly required in order to establish a classification of failure types with similar consequences to simplify robustness assessment. From the perspective of the electrical subsystem Lu et al. [12] and Edomah at al. [7] give an overview of the most common power quality problems (e. g., voltage sags) in power systems including their consequences and possible technical mitigation. However, particularly the description of consequences is only generally described and details are missing how consequences could be analysed or modelled specifically for given smart grid installations. The interdependency description model developed in this work, however, takes topology and structures of a given power system into account. The most available work corresponding to grid failures can be found in the area of the analysis of cascading failures. A commonly followed approach is to model the electrical power grid as a graph, where transformers, loads and generators are represented as vertices and transmission lines as edges. Failure analysis on top of such models is then carried out mainly in two flavours. While the first approach only analyses the structure of the graph (e. g., node degrees) to derive general insights into consequential spread and speed of failure cascades induced by natural events of attacks (e. g., [4] [24], [11]), the second approach also respects physical power flows in the electrical grid (e. g., [6], [14]). By employing power flow simulations the propagation of failure cascades is evaluated more quantitatively compared to the first approach. Another example is the approach of Koç based on modelling the electricity grid, where “power grid is considered as a complex network in which electricity is exchanged between nodes through the shortest or most efficient path” [10]. This work not only focuses on cascading failures within the electrical subsystem but additionally considers the ICT subsystem. Specifically, failure propagation paths between the electrical and the ICT subsystems are being modelled in order to describe mutual effects between both subsystems onto the power grid robustness. In this direction of interdependencies between the ICT and the electrical subsystem, a variety of research has been conducted recently. Analysis methods for interdependency analysis range from empirical approaches, simulative approaches to analysing system dynamics and economic theory towards network-based approaches [16]. Par- ticularly for power systems, network-based approaches have been more often proposed. E. g., the approach based on complex network theory [5] which is applied to interdependencies between power systems and ICT. Also, complex systems approaches have been proposed, e. g., using state machines representing if ICT or grid components operate correctly and how they affect each other [23]. However, these approaches operate mainly on a qualitative analysis not trying to quantify the effects those interdependencies might have on the overall system output if one or more components get unavailable. By using simulations and machine-learning, the interdependency model developed in this work will allow for the transition from purely qualitative evaluations towards quantitative evaluations. 16 Robustness Modeling and Assessment of Interdependent Smart Grids 215

The robustness of an electrical grid is quantified by measuring damage due to a decrease in network efficiency if certain nodes of a network become non-operational [6]. Another approach assesses network robustness with respect to cascading failures for different tolerance parameter values in targeted and random failure scenarios [9]. Another approach quantifies the robustness of a power grid by using network theory (namely giant component theory) [14]. Furthermore, Bao et al. considers “the entropy of global load distribution as an indicator of the size of cascading failures” [2]. Youssef et al. are concerned with quantifying network robustness by the probability of link survivals resulting from cascading failures [25]. The application of machine learning approaches has been proposed to analyse and evaluate the risk and consequences of cascading failures within power grids, for instance, by applying a Bayes network learning implications of various operating conditions (random loads) of the power grid [17]. Another ML-based approach trains a Support Vector Machine with simulated line trips leading to cascading failures due to overload on remaining lines. The order of lines affected by the failure cascade is chosen by employing probabilities and the properties of probability distributions [8]. Recently, work has started to apply Organic Computing concepts into the field of smart grids. For instance, an architecture for OC systems following the observer-controller patterns have been proposed as a means for equipping complex systems with self-organising capabilities [20]. In this context, smart grids have been named as a particular field of application. Furthermore, OC concepts have been proposed for application in energy management systems [13]. In the case of machine-learning applications for power grid robustness analysis, there seems to be a lack with respect to adequately consider interdependencies between the subsystems - the topic tackled in this work.

16.4 Methodology

The methodological approach to the sub-research questions is explained in the following. For answering RQ1, literature research is necessary in order to get an overview of failures potentially happening in the considered electrical supply system. Particularly failures, which are subject to interdependencies between the electric and the ICT system are in focus of examination. The goal is to provide classes of failures at component level, which have comparable effects on power supply quality. For that, a top-down or a bottom-up approach can be chosen. While the top-down approach looks at the final effects such as voltage deviation or power deficiency of grid regions and follows the chain of incidents down to grid / ICT component level. The bottom- up approach analyses potential failures at components such as line overload, outages of sensors following the line of incidents up to the potential results onto power supply quality. As it can be considered that there is a huge diversity of component failures and potential effects on to power supply, in a first step most relevant grid failures (in distribution systems) are selected from the literature. 216 F. von Tullenburg¨

RQ2 delivers a general and transferable model of interdependencies between the ICT and the electrical subsystem of the electrical grid. The interdependency analysis focuses mainly on physical dependencies from the electrical system to the ICT system and cyber dependencies from the ICT system to the electrical system. While physical dependencies are due to the necessary power supply of ICT devices, cyber- dependencies arising from the transmission and processing of sensor and control data. Thus, it is started with a structural analysis of the system is pursued considering the topological mapping of grid components, ICT components and electrical supply regions (based on the switching state of the grid).

A2 P1 P SR2 SR1 0 T1

I2 I3 I5 D2 D3 I4

D4 R C1 1 D5 D0 D7 P2 D 1 D6 R0 D17 SR3 A3 I6 R D10 2 D9 D D8 13 P3 D16 D11 C0 D14 I1 D12 R3 I7 I I0 D15 I8 T 9 3 P 4 P A1 5 T2 A0 P6

Fig. 16.1: A smart grid architecture consisting of 3 (switching-state dependent) regions, ICT components and grid components.

An exemplary distribution system (based on a real-existing distribution grid) is depicted in Figure 16.1. The electrical system consists of three transformer stations (T1 - T3), connected to a system of power lines (P0 - P6), line switches (A0 - A3) and busbars, where power lines are interconnected. In distribution networks these busbars are usually installed in closed compartments at switchgears. The division of the electrical network in supply regions is determined by the state (opened/closed) of the line switches (switching-state). In the depicted case, there are three supply regions (SR1 - SR3), where SR1 is supplied by transformer T3, region SR2 is supplied by T1 and region SR3 is supplied by transformer T2. At the ICT side, the power system is equipped with several intelligent electronic devices (IEDs) denoted with I0 - I9. These IEDs are either sensors or actors capable of measuring electrical parameters or activate a power switch. In particular, IEDs are connected to the grid monitoring and control systems located at the central control stations (C1 and C2) over the ICT network comprising of forwarding devices (or routers) (R1 - R4). Usually, this ICT network is usually designed as a ring topology. Data links are depicted as green lines, where solid lines represent active links and dashed lines represent inactive links in the current conﬁguration. 16 Robustness Modeling and Assessment of Interdependent Smart Grids 217

D16 D7 D6 D17 D5 T1 D3 T2 D2 S0

C2 T1 S1 I0 S2 I8 S3 I7 I3 I6 I4 I5

Fig. 16.2: Direct physical (solid) and cyber (dashed) dependencies control station C2 has on other components in the current situation in the considered system.

The structural analysis approach uses a systematical approach for constructing a dependency graph from the topological structure of the power grid. For each component in one of the subsystems, it is derived, from which other components it is function- ally dependent (due to cyber and or physical dependencies). For this, direct dependencies between any component and other components are analysed. In this context, direct means that a component is immediately dependent on the output of another component. The dependency graph showing direct dependencies for central control station C2 is depicted in Figure 16.2. The graph based on the example depicted in Figure 16.1 and shows the dependencies of component C2 (a control system) on other components. The component denotations are taken from Figure 16.1. Dashed lines represent cyber-dependencies between component C2 and other components, while solid lines visualising physical dependencies. Second, the individual dependency graphs for each component can be combined with a complete dependency graph also representing indirect dependencies. Figure 16.3 shows the resulting fully combined (potentially cyclic) dependency graph. For the sake of clarity and simplic- ity, this ﬁgure shows only a small part of the dependencies that actually exist. Generally, for each component, a dependency graph with direct dependencies can be derived and afterwards combined to the fully combined dependency graph. For this approach, we oriented on literature which already proposed similar descriptions for interdependencies. A contribution that goes beyond the state of the art will be a method of partially performing the structural analysis automatically. It is envisioned to use information available from energy management systems (EMS) and network management systems (NMS). Information on the current switching state can be used to derive physical dependencies with respect to the power supply of ICT devices, while information on active communication network paths is used to derive cyber dependencies between ICT devices. In addition, expert knowledge can reﬁne the mo- 218 F. von Tullenburg¨

S3 T2

Fig. 16.3: Section of the dependency graph resulting from combination of individual dependency graphs. del with additional information not included in management systems. Although this analysis method is required for later steps, in the course of the project this analytical method is expected to be developed on a conceptual level rather than producing an operable and integrated analysis system. The work on RQ3 elaborates on developing the machine-learning based approach for operational robustness assessment of smart distribution systems. Methodologically, a distribution system simulation will be developed also including the corresponding ICT infrastructure as co-simulation or integrated simulation. It is the plan to use a simulative approach to evaluate how different failure scenarios (resulting from RQ1) will affect the quality of power supply of the power system. For certain incidents appearing at one component of either subsystem, it can be evaluated how they affect other components including spread into the other subsystem. The simulation scenarios examine different subsystem configurations resulting in different information flows and power flows in order to derive changes of interdependencies. The determined effects are evaluated and used for quantifying the strength of dependencies. For the purpose of quantifying the impact of cascading failures, the simulation environment includes a generic model for evaluating the deviation quantitatively from optimal grid conditions (which is a result of RQ2). Intuitively, an electrical grid is in optimal condition, if the utilisation of grid equipment (transformers, transmission lines, etc.) is equal for all components. Additionally, with respect to robustness against cascading failures, we assume, that the node degrees, representing the number of incoming and outgoing dependencies are also equally distributed, avoiding the occurrence of hubs which are affected by many other components or affect many components if itself fails. For both subsystems simulation models already available from literature might be applied. The dependencies depicted in the graph representation will be assigned normalised values (e. g., between 0 and 1) representing the impact a cascading failure going along that dependency will cause. The impact ex- presses how far (from a topological point of view) a failure will spread and how 16 Robustness Modeling and Assessment of Interdependent Smart Grids 219 strong the deviation from optimal grid conditions is. While failure spread can be derived from topological information the condition deviation requires in particular load flow computations to be executed by the simulation.

ICT Grid Topology Topology

Dependency Expert Analysis Knowledge

Dependency Model

Simulation Environment

ICT Failure Grid Failure Scenarios Scenarios

Eects Eects

Eects on Failure Cascade Simulation Eects on Grid & Learning ICT

Power-Flow Grid Voltage Analysis Control Simulation Models

Real-Time ICT Learning Real-Time Grid Data Results Data

Classier

Operational Robustness

Fig. 16.4: Procedure of robustness assessment approach starting from initial grid modelling, over simulation and classiﬁer training to operational robustness assessment.

However, at this point, valuation and quantification are subject to specific system configuration and utilisation and the results potentially lack generality. As it is not a viable way to simulate every possible combination of system configuration and system state for every failure scenario of interest, machine learning comes into play. The results of the simulative analysis provide input to train a classifier system (e. g., Support Vector Machine - SVM) capable of mapping grid configuration and state to a risk metric for specific failure scenarios. In the last step, the trained classifier is used for online evaluation based on current, possibly before unseen, boundary conditions. Thus, a main aspect of the ML approach will be its capability to compare different situations with each other and evaluate a given (currently unknown) situation in 220 F. von Tullenburg¨ comparison to similar (possibly simulated) situations. By comparing similar simulated scenarios, the ML system might also be able to extract how particular changes in state or configuration affect overall system robustness. Another approach would be to create and evaluate a new simulation scenario based on the current situation, potentially using an already existing similar simulation scenario as a starting point in order to short evaluation time. The proposed approach is shown in Figure 16.4.

16.5 Conclusion and Outlook

In this work, an approach for analysis and evaluation of power supply system robustness with respect to interdependencies between ICT and the electrical subsystem is proposed. A three-step approach is envisioned: (1) the inﬂuences of interdependencies on power grid robustness is analysed, (2) a model for describing and quantifying interdependencies is developed, (3) a machine learning approach is developed help- ing to analysing system robustness based on current system state information. The machine-learning approach is pursued in order to achieve a fast analysis to be used for online assessment. The main contribution of this project is to allow power grid operators to steadily examine the resilience of the electrical system with respect to failures, which could propagate and cascade between both subsystems induced by unpredicted events. Furthermore, the developed approach could provide a viable basis for autonomic grid management systems, based on Organic Computing concepts. The next step in course of the project is the detailed analysis of potential effects of interdependencies onto the power grid robustness. Building upon this, a model for describing interdependencies and their risks is developed.

References

1. Andersson, G., Donalek, P., Farmer, R., Hatziargyriou, N., Kamwa, I., Kundur, P., Mar- tins, N., Paserba, J., Pourbeik, P., Sanchez-Gasca, J., Schulz, R., Stankovic, A., Taylor, C., Vittal, V.: Causes of the 2003 Major Grid Blackouts in North America and Europe, and Recommended Means to Improve System Dynamic Performance. IEEE Transac- tions on Power Systems 20(4), 1922–1928 (Nov 2005) 2. Bao, Z., Cao, Y., Wang, G., Ding, L.: Analysis of cascading failure in electric grid based on power ﬂow entropy. Physics Letters A 373(34), 3032–3040 (Aug 2009) 3. Bellman, K.L., Tomforde, S., Wurtz,¨ R.P.: Interwoven systems: Self-improving systems integration. In: Eighth IEEE International Conference on Self-Adaptive and Self- Organizing Systems Workshops, SASOW 2014, London, United Kingdom, September 8-12, 2014. pp. 123–127 (2014) 4. Bhave, A.S., Crow, M.L., Cetinkaya, E.K.: Robustness of power grid topologies against centrality-based attacks. In: 2016 Resilience Week (RWS). pp. 115–118. IEEE (Aug 2016) 16 Robustness Modeling and Assessment of Interdependent Smart Grids 221

5. Buldyrev, S.V., Parshani, R., Paul, G., Stanley, H.E., Havlin, S.: Catastrophic cascade of failures in interdependent networks. Nature 464(7291), 1025–1028 (Apr 2010) 6. Crucitti, P., Latora, V., Marchiori, M.: Model for cascading failures in complex networks. Physical Review E 69(4) (Apr 2004) 7. Edomah, N.: Effects of voltage sags, swell and other disturbances on electrical equipment and their economic implications. In: CIRED 2009 - 20th International Conference and Exhibition on Electricity Distribution - Part 1. pp. 1–8. IET (2009) 8. Gupta, S., Kambli, R., Wagh, S., Kazi, F.: Support-Vector-Machine-Based Proactive Cascade Prediction in Smart Grid Using Probabilistic Framework. IEEE Transactions on Industrial Electronics 62, 2478–2486 (Apr 2015) 9. Kinney, R., Crucitti, P., Albert, R., Latora, V.: Modeling cascading failures in the North American power grid. The European Physical Journal B 46(1), 101–107 (Jul 2005) 10. Ko, Y., Warnier, M., Kooij, R.E., Brazier, F.M.: An entropy-based metric to quantify the robustness of power grids against cascading failures. Safety Science 59, 126 – 134 (2013) 11. Lai, Y.C., Motter, A.E., Nishikawa, T.: Attacks and Cascades in Complex Networks. In: Ben-Naim, E., Frauenfelder, H., Toroczkai, Z. (eds.) Complex Networks, vol. 650, pp. 299–310. Springer Berlin Heidelberg, Berlin, Heidelberg (Aug 2004) 12. Lu, Z., Lu, X., Wang, W., Wang, C.: Review and evaluation of security threats on the communication networks in the smart grid. In: 2010 - MILCOM 2010 MILITARY COMMUNICATIONS CONFERENCE. pp. 1830–1835. IEEE (Oct 2010) 13. Mauser, I., Hirsch, C., Kochanneck, S., Schmeck, H.: Organic Architecture for Energy Management and Smart Grids. In: 2015 IEEE International Conference on Autonomic Computing. pp. 101–108. IEEE (Jul 2015) 14. Motter, A.E., Lai, Y.C.: Cascade-based attacks on complex networks. Physical Review E 66(6) (Dec 2002) 15. Muller-Schloer,¨ C., Tomforde, S.: Organic Computing – Techncial Systems for Survival in the Real World. Autonomic Systems, Birkhauser¨ Verlag (October 2017), iSBN: 978- 3-319-68476-5 16. Ouyang, M.: Review on modeling and simulation of interdependent critical infrastructure systems. Reliability Engineering & System Safety 121, 43–60 (Jan 2014) 17. Pi, R., Cai, Y., Li, Y., Cao, Y.: Machine Learning Based on Bayes Networks to Predict the Cascading Failure Propagation. IEEE Access pp. 1–8 (2018) 18. Tomforde, S., Hahner,¨ J., Sick, B.: Interwoven Systems. Informatik-Spektrum 37(5), 483–487 (2014), http://dx.doi.org/10.1007/s00287-014-0827-z, Aktuelles Schlagwort 19. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Mnif, M., Muller-Schloer,¨ C., Richter, U., Schmeck, H.: Observation and Control of Organic Systems. In: Muller-¨ Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing-AParadigm Shift for Complex Systems, pp. 325 – 338. Autonomic Systems, Birkhauser¨ Verlag (2011) 20. Tomforde, S., Rudolph, S., Bellman, K., Wurtz,¨ R.: An Organic Computing Perspective on Self-Improving System Interweaving at Runtime. IEEE International Conference on Autonomic Computing p. 9 (2016) 21. Tomforde, S., Sick, B., Muller-Schloer,¨ C.: Organic Computing in the Spotlight. arXiv.org (January 2017), http://arxiv.org/abs/1701.08125 22. Union for the co-ordination of transmission of electricity (UCTE): Final Report - System Disturbance on 4 November 2006 (2007) 23. Waﬂer,¨ J., Heegaard, P.E.: Interdependency modeling in smart grid and the inﬂuence of ICT on dependability. In: Meeting of the European Network of Universities and Com- panies in Information and Communication Engineering. pp. 185–196. Springer (2013) 222 F. von Tullenburg¨

24. Wang, J.W., Rong, L.L.: Robustness of the western United States power grid under edge attack strategies due to cascading failures. Safety Science 49(6), 807–812 (Jul 2011) 25. Youssef, M., Epicenter, K.S.: Robustness measure for power grids with respect to cascading failures. Proceedings of the 2011 International Workshop on Modeling, Analysis, and Control of Complex Networks p. 5 (2011) 17

Survey on Urbantrafﬁc Control and Management Using Multi-Agent Systems and Model Predictive Control

Lena Elisa Schneegans

Trafﬁc Engineering and Transport Logistics, University of Kassel, Kassel, Germany [email protected]

Abstract. Intelligent trafﬁc control systems are becoming increasingly important e. g. regarding the number of motorised road users, which is growing faster than the capacity of trafﬁc routes. This paper surveys urban congestion management using multi-agent systems and model predictive control. Results are discussed and presented in Tables.

Keywords: Congestion, Flow Optimisation, Intersection Control, Intersection Management, Model Predictive Control, Multi-agent Systems

17.1 Introduction

Intelligent traffic control systems are in discussion of becoming the new technology for congestion management whereass the number of motorised users is growing faster than the capacity of traffic routes. So traffic is flowing more slowly. Where in Germany 1960 73 million people and only 4.5 million vehicles moved, today 82.5 million people move in 46.5 million vehicles [46]. The length of the road transport network after reconstruction after the 2nd world war grew less than the above described issues [46]. Moreover, the number of commuters has steadily increased as well as the distance covered to get to work [14]. This pushes trends like increasing automation of vehicles and demands on environmental protection. To support these developments it is advisable to coordinate the different elements of smart city networks such as vehicles, traffic lights and other sensors with each other in order to maximise the effectiveness of the separate parts and deliberate control effects of their interaction in a common system. This survey is intended to provide an insight into the current results in the field of intelligent transport systems for traffic flow control caused by intersections in the urban road network. The focus lies on distributed approaches with multi-agent 224 L. E. Schneegans systems and model predictive control for the coordination of traffic junctions. This work addresses the following questions: • What are the general challenges and requirements facing intelligent transport systems, especially for road traffic in cities? • Which solutions does Artificial Intelligence (AI) offer, in particular organic computing multi-agent systems (MAS) for the optimisation of traffic flow and reduction of traffic congestion to tacle real world urban traffic? • Which approaches offer model predictive control (MPC) for the optimisation of traffic flow and reduction of traffic congestion? In Section 17.2 general information on intelligent transport systems for urban intersections is presented to give an orientation to identify requirements. Section 17.3 offers a discussion based on the found references, which are briefly introduced in Tables. A conclusion is given in Section 17.4.

17.2 Foundations

Road transport networks are mostly public property, therefore their organisation is in public hands. Consequently, decisions on implementation of intelligent transport systems (ITS) are reached on the level of local government and implemented as well as maintained by the relevant public executive bodies. These parties usually act conservatively due to e.g. operational times of the used technical systems of over 20 years and sales monopoly by Siemens and Swarco with established solutions. Thus, the challenges mentioned above are heard and discussed by politicians on all levels. The UN and EU have issued directives which aim to guide future developments for intelligent transport systems and smart cities [31]. Concepts of ITS are used to designate integration of control, information and communication technologies within transport infrastructure [47]. In the future, we do not expect our roads to be used exclusively by autonomous vehicles. Road users, such as children, cyclists, and people-controlled vehicles, that behave in a non-deterministic way, will continue to participate in road traffic. This is a particular challenge for traffic planning and control, as all road users must be able to be reliably safe coordinated. In order to understand the field, it is advisable to look at the related context. The system requirements are complex. Most important is the acceptance of the system by users and integration of their typical actions such as acceleration behaviour and distance control [45], [4]. Transportation is hereby seen as the enabler for all other services in public infrastructure [45]. Therefore, the development of urban transport networks should take up current social trends such as individualism and automati- sation and counteract possible negative effects. Challenges for such a system are, e.g., user acceptance of advice given by network data services, individualism, and diversity of users and devices as well as their interests. Further attendance is needed 17 Survey on Urbantraffic Control and Management 225 for the integration of groups of interest like elderly or disabled people, emergency services, the point of public interests challenging individualism and fairness between device owner interests and technologically non-equipped users [45]. Road transport networks of cities can be viewed as a whole, but also separately via isolated intersections or segments of the network. Thus, questions can be taken from the micro up to the macro perspective. Micro perspectives are for example autonomous driving with or without regard of interaction with other participants. There, decisions are made about the driving behaviour and the choice of route, e.g., to optimise the driving time. Macroscopic perspectives are the overall control of complex networks, where more attention is paid to platoons or traffic flows optimising total waiting times or average speeds [47]. A concept fusing different fields of research in ITS is given in [53]. The paper presents the concept of so-called parallel transport management systems, which tries to fuse findings from artificial intelligence, complex theory, social computing, agent theory, etc. According to the possible perspectives between micro and macro, different architectures are developed for communication networks in smart cities. There are three ways of structuring ITS: central, hierarchical and distributed. In central structures, decisions are made by one element, like a central command center, in one location for all elements of the system involved, such as traffic lights, vehicles and people. In distributed systems, however, decisions are made by each element, e.g., traffic lights or vehicles, itself. If necessary communication to or information about other elements is used. In hierarchical systems, decisions are made at different levels. In addition to the decisions left to the individual elements, there are superordinate decision-makers who make decisions for groups or the entire network. Hierarchical and distributed Systems can well be mixed. There are some arguments in support of (hierarchically) distributed systems: they are more robust against disturbances due to their local impact, more reliable in case of failure of decision units, because some keep working and usually more real-time capable due to the distribution of the problem to several computers. Solutions archieved by distributed systems can however no longer be regarded as optimal from the point of view of optimal control. But they usually achieve nearly optimal results, see e.g. for freeway control [18]. The authors of [45] present a concept called autonomic transport management system (ATMS). A network structure including different hierarchical levels is shown. The ATMS is set in close relation to the actual political requirements given by UN and EU, including established standards, see [3]. The concept aims for a network structure collecting and providing data via services in a back end. This network is connected with autonomic systems, e.g., on board units in cars and applications for smart phones. The cooperating devices make their own decisions relying on the network data. The connection between the backend and the on board units is given in [58]. Decision processes in road traffic done by infrastructure as well as users/ autonomous cars are divided into three types: long term (strategic), medium (tactical) and short (operational). Long term addresses questions of when, how and with what 226 L. E. Schneegans to travel. Medium term answers questions of path planning, where along. And short term treats decisions in the actual moment, like line change, overtaking or path choice at intersections. The key is hereby seen in influencing travelers behaviour by giving tactical information. Important terms for communication in ITS are communication from vehicle to vehicle V2V, vehicle to infrastructure V2I and in a mixed fashion including other technical devices V2X [11]. An overview on cited references in this section can be found in table 17.1. Identified requirements for an ITS controller are: • Ergonomic design, acceptance of the system by the users • Ergonomic design, acceptance of the system by maintenance department and political actors • Environmental protection • Reduction of travel time and other positive travel arrangements, such as energy consumption, a path through calm areas • Interests of individual but also community, current social trends • Diversity, integration and tolerance of all potential road users • Safety and fairness to participate safely in the system regardless of the level of technology used • Decision (support) in real time • Dynamic integration and segregation of agents in a system under control, respective of their diversity • Communication capability • Predictive awareness of consequences and coming situations • Integration of valid standards

Table 17.1: Table of references on foundations

Ref. Field of Interest Problem Solution [47] ITS, general problems and perspectives - (book) [45] ITS, smart cities autonomic transport IT-network / management systems framework / services [31] ITS, management and getting to ”Green Transport” political committee political point of view directives [58] ITS, smart cities congestion service packages tool [4] ITS, human factor human behaviour - (book) [3] ITS, Communication road safety - (book) [53] ITS, engineering and social transport control and diverse concept fusion; management Input: Network and Sensor Data; Evaluation variable: average vehicle speed [11] intersections Cooperative intersection - (survey) Management 17 Survey on Urbantrafﬁc Control and Management 227 17.3 Discussion

This section presents the results of the literature research. The literature is discussed and reflected on the filed of application. Thus it is not predented in an direct way, although short descriptions can be found in Tables 17.1, 17.2, 17.3, 17.4 and 17.5. Initially, the discussion will address overlapping aspects of all three fields MAS, AI and MPC. A classification of the methods of MAS, AI and MPC in the field of application of urban traffic can be found in the corresponding subsections. Special aspects in this respect are also briefly addressed there. The field of application poses challenges for future developments. Thus, a number of challenges for urban transport networks are identified in [31], for example in reliability, comfort, parking space, congestion and cargoes. Further challenges arise from the characteristics of the network. Transport networks generally are geographically distributed systems in which the participants make decisions as individuals, which shape the state of traffic. These decisions depend on the information available about the system, e.g. about the presence and behaviour of participants in the vicinity. The current regulations for light signal systems function similarly. Their switching behaviour depends on the detected incoming vehicles as well as on fixed procedures. The according properties are highly dynamic and require extensive flexibility of the models [52]. This flexibility can be given in various ways. In the area of MPC, for example, adaptive or hybrid models are used or a part of the network, that can be well modelled, is used for control and the rather unpredictable dynamics of the environment including human behaviour and thus disturbances are attributed. Applications from MAS and AI proceed similarly and choose suitable scenarios for the application of their theories. Those scenarios partly directly address the prediction of such environmental conditions and dynamics. With reference to the real application, further challenges arise, such as incomplete information and mixed groups regarding the degree of technisation [56], real-time capability of algorithms and handling emergencies [17] and privilege of public transpsort. Further the integration of so called vulnerable road users (VRU) such as pedestrians and cyclists is of interest [11]. In view of these challenges, there is a need for research. A variety of measures in research and development to approach urban traffic challenges can be emphasised: for vehicle platoons, public transport and decision support systems, of which all rely on parameter optimisation based on network data, e.g. analysed by artificial intelligence algorithms [31]. The prediction of network dynamics is also an important point [45]. Most researchers look for scenarios to illustrate their theories. Normally, these representations do not go beyond simulations. There is a lack of results with a reference to reality. Although the real-time capability of the algorithms used is discussed. A relation to the developments and possibilities of the real existing networks is rarely given, especially with regard to the references on MPC. Thus, with a few exceptions, norms, standards and socio-political interest groups are excluded, although these are essential for practical application. 228 L. E. Schneegans

In addition to this lack of applications, there is a lack of strategic decision algorithms that exploit the potential of the detected data. Besides also the comparison between the different solutions is difficult, see [13] in [27]. The reason for this is the pronounced diversity of approaches and use of methods. Transitions between iden- tifiable sub-areas are fluid. Possible structural subdomains are: autonomous driving, individual junctions, intersection networks and data networks as well as variables considered: path, flow, time and position. Existing traffic simulation tools are rarely used for simulations, some of which come very close to the real human decision-making process. Vissim [16] should be mentioned here as the tool used in case. Instead, mostly either in-house developed frameworks or familiar tools from the respective method domain are used. With regard to MAS, the use of JADE [7] attracted attention. Another framework for multi-agent systems is ALICA, which also has been tested for traffic szenarios, [38]. This trend hinders a user-oriented approach and a system design that takes ergonomic aspects into account. Another point regarding simulations are the underlying networks. For seperately addressed intersections, right-angled intersections with single-lane roads in each direction are often used. As soon as several intersections in a network are considered, there is a shortage of comparable structures. Standards from the research areas of transport have not yet gained prominence in the research on MAS and MPC, which mostly stems from computer science or electrical engineering. A similar diversity can also be observed with regard to control and analysis variables. They can be broken down according to the above-mentioned sub-areas: path, flow, time and position. The prediction of routes taken, path planning and decision support services for route decisions usually takes the macroscopic view of the entire network. Views looking at quantities of flow, density, capacity and queue length can be found in approaches ranging from the macroscopic to the microscopic view. What is noticeable here is a rather heterogeneous naming of different data. For example, the queue length can mean the number of waiting participants at an intersection, but also the actual length of the queue in meters. Most variables measure time. It is used in maxima, sums, averages, etc., preferably in the form of total travel time, waiting times or delays and crossing times at single intersections. The most uniform use can be found in the sub-area of autonomous driving: position, speed and acceleration are used here to control. However, the evaluation of the control quality is again subject to creativity within the scope of the task, leaking comparability. As current research trends, the already mentioned sections can be named: autonomous driving, separate junctions, intersection networks and data networks. The research directions of the overlapping branches are also heterogeneous. For intersections with high traffic volumes, for example, solutions with traffic lights are compared with those without. The adaption of traffic ligh control based on vehicle parameters e.g. in [40], [54] is an excample for cooperative ITS, what is reviewed in [22]. In [11] future research is argued between traffic lights with algorithms considering platoons or groups of interest led by e.g. speed waves with models for trajectory modelling, collision regions and coordination methods especially based on multi- 17 Survey on Urbantraffic Control and Management 229 agent methods and non-signalled intersections, which are rated as a more promising future solution for intersection management. An extensive field of research in transport technology is autonomous driving. Most multi-agent system and MPC references focus on research in this area in the context of industrial interests and environmental protection and dust avoidance at traffic light junctions. As these are all directly related to motorised vehicles, other road users such as pedestrians and cyclists are left out of the picture. Other fields such as communications engineering already include them in their research, see [11]. Although most MAS for this purpose are based exclusively on networks of autonomous vehicles, the authors of [10] emphasise that it is possible to integrate vehicles with human control into such networks.

17.3.1 Multi-Agent-Systems and Artiﬁcial Intelligence

The following paragraphs focus on specific aspects of AI particularly MAS in the field in scope. Transport networks are geographically distributed systems in which the participants make decisions as individuals. These decisions depend on the information available about the system, e.g. about the presence and behaviour of participants in the vicinity. The current regulations for light signal systems function similarly. Their switching behaviour depends on the detected incoming vehicles. On this basis individuals and traffic light systems can be regarded as interacting agents in a distributed system, from the point of view of agent theory, so-called Multi-Agent Systems (MAS). Explanations on MAS are dispensed with at this point and a basic understanding of MAS is assumed. The mentioned references can be found in Tables 17.2 and 17.3. According to the authors of [10], MAS applications are widely used. Agent theory is ideal for traffic and transport, since dynamic, rapidly changing traffic conditions are combined with requirements for modularity and adaptability. In addition, transport networks are complex and often unstructured, at least at the beginning [13]. Due to its relatively young age, there is a high diversity of methods in this field of research. Most approaches from multi-agent theory also use other research areas, such as MPC or artificial intelligence such as neural networks, learning strategies and organic computing. Often without explicit mention of agents. Agent-based systems for road traffic usually consider autonomous driving and decentralised decision support. How- ever, there are other fields of interest. So in [6], e.g., the development of information strategies for dealing with Braess Paradox is addressed. Braess Paradox from 1968 deals with the question of adding roads to networks and the issue that this does not necessarily reduce congestion. Or in the paper [5] the authors describe a prediction principle for traffic congestion. An important point is that the agents trained off-line in smaller networks also reach the goal of predicting traffic congestion in unknown larger networks [41]. 230 L. E. Schneegans

Another field is being jointly investigated by researchers from MAS and Commu- nication systems. Smart Cities are built on an extensive basis of sensor data. Ar- chitectures and frameworks are required for preparation and structuring for further processing. This concerns the macroscopic level of urban transport networks. How- ever, the research of these architectures goes one step further and suggests solutions down to the smallest microscopic questions. Therefore, they are usually structured hierarchically. 17 Survey on Urbantraffic Control and Management 231 grading Variable - - - - average travel time flow, decisions flow, decisions link capacity average green light percentage average delay Control Variables - - - - route information flow, decisions flow, decisions network information, keep or change track optimising waiting time, total network time and waiting time at the intersection to control vehicle acceleration amount of approaching vehicles Solution - (survey) - (book) - (survey) construction guidelines / architecture route information sharing communicating subnetworks, continuous, discrete and integer variables communicating subnetworks, comparison between serial and parallel coordination schemes neuronal network adaptive multi-objective reinforcement learning reservation scheme Table 17.2: Table of references on MAS Problem architectures, platforms, autonomous driving, decision support smart cities, congestion logistics decision support systems congestion traffic flow control traffic flow control congestion traffic signal control intersection control Field of Interest MAS application in traffic and transport systems applications of agent technology in traffic and transportation MAS for transportation and traffic management Traffic Management route guidance mechanism large-scale transportation networks large-scale transportation networks prediction of traffic conditions cooperative multi-agent framework traffic management Ref. [10] [27] [13] [39] [55] [35] [36] [5] [26] [15] 232 L. E. Schneegans grading Variable autonomy, distributed functioning, self-organising, potential adaptability and robustness cost function average vehicle speed average delay waiting time average vehicle delay - - - Control Variables - information traffic flow volumes; distribution, variability, capacity of the road trafficthe in network different traffic dynamics, queue length, commulated delay, arrival rate traffic flow, density traffic flow - - floatin car data, traffic light times Solution swarm intelligence information manipulation strategy Bayesian and Neuronal Networks reinforcement-, Q-learning game theory organic computing, adaptive learning, coordination mechanism with neighbouring intersection support vector machines - (Review) vehicle parameter estimation of vehicle Table 17.3: Table of references on AI Problem authentic modelling Braess Paradox prediction adaptive traffic sign control multiple intersections traffic light control Prediction of traffic lights Potential Effects of Data Interchange V2I adaptive traffic light control Field of Interest transport modelling information strategies ITS, traffic flow urban traffic smart transport urban road network ITS ITS cooperative ITS Ref. [49] [6] [41] [52] [8] [43] [54] [22] [40] 17 Survey on Urbantraffic Control and Management 233

17.3.2 Model Predictive Control

This section provides a brief explication of the idea of model predictive control and references for further understanding. Followed by application issues of MPR to traffic management, see therefore also Tables 17.4 and 17.5. Model Predictive Control (MPC) is one of the regulatory procedures widely applied in the industry. An introduction including different forms of MPC is given in [32]. MPC controls a system using a system model that predicts the system behaviour and its states x for the future. This prediction is reformulated into an optimisation problem with boundary conditions. The Solution of the optimisation is the next control variable u. Prediction takes place at any time instant over a prediction horizon HP in j steps. The optimisation problem is usually formulated using a cost function. In the classical case, in a cost function a deviation of system variables from defined references (xR,uR) is punished. The deviation is minimised with the aim to approximate system variables to their references [29]. The advantages of this control method are in particular the intuitive approach, the high flexibility for modelling and the design of the cost function as well as the possibility to consider variable restrictions. With respect to distributed MPC [12], [9] and [34] provide an overview. The authors of [44] give an insight into architectures for distributed MPC. The possibilities to use MPC distributed are extensive due to the multitude of design freedom, like the choice of model, formulation of the optimisation problem and assignment of shared information variables and equal design of different agents, see here for examples [12], [9] and [44]. In general, references using MPC for traffic control can be divided into two groups. The first group deals with central control and management of intersection networks and the second with decentralised intersection management with consideration of intersection participants. Decentralised approaches that consider interactions between crossings were found only in [48]. The interest in the exemplified control of transport networks with the help of MPC is based on the trend in theoretical research on large scale systems of all sorts. A special challenge lies in the non-linear dynamics of such a network. Aspects of stability can, however, be easily answered here, depending on the choice of variables. It is a naturally stable system in so far as traffic flows, capacities and the like are considered, as roads of these sizes are subject to natural limits that can be included in the MPC [48]. In terms of microscopic vision, autonomous driving is also a popular field of research here. In particular, the interaction between vehicles and also with the infrastructure are a focus here. Besides, research on interactions between crossroads is an upcoming field [19, 43, 42, 51]. The decision between centralised or decentralised control is often discussed [50]. It is often argued that only centrally controlled algorithms can achieve optimal results, but almost optimal solutions with a higher reliability and decision frequency can 234 L. E. Schneegans be reached by distributed approaches. For example, the distributed solution in [37] converges to the global optimum claiming that for a realistic scenario.

17.4 Conclusion

In this survey, an insight is given into the topic of traffic flow control in urban areas with special focus on approaches of multi-agent systems and model predictive control. All in all potentials as well as challenges could be identified. The analysis of the references has revealed extensive potentials. MAS and MPC both suit the technical challenges of system dynamics and interferences, especially to tacle in environmental protection and user comfort such as travel-time-reduction. Strengths and weaknesses of these methods complement each other. The use of AI and machine learning can give additional improvement to such methods. Because of the high dynamics and complex interactions transport systems are hard to model. AI reduces the effort for modelling. But also weaknesses of these methods were found, measured on the listing in Section 17.2: • System acceptance of network ”owners” (local politicians and maintenance departments). • Integration of human behaviour, including social trends. • Integration of diversity of users and technical systems in use. • Leak in guaranteeing safety with respect to intersection signalisation and softwarefailure. • Real time capability. Accordingly, further research is recommendable on these methods to make them applicable to real world street networks. Futher research can go in directions of: • Comparison and evaluation of the field methods presented including standards. • Adjustment of existing methods towards real world application. • Integration of diversity of users. • Further exploitation of detected data in algorithms. In order to achieve the applicability of the methods under consideration, aspects of user acceptance, norms and standards and also political objectives should be given more attention. Also, it is currently difficult to compare algorithms with each other. Standards for simulation tools, scenarios and analytical parameters are expected to emerge in the future. The discussed methods come from the fields of control and AI in electrical and mechanical engineering and computer sciences. Ordinarily, the different disciplines in engineering do not seem to discuss intensively with each other. There is need of 17 Survey on Urbantraffic Control and Management 235 grading Variable total journey time, waiting time total travel time per vehicle, average speed, delay time and number of stops per vehicle - total time spent, total emissions number of vehicles, flow times to cross, leave the area position, velocity Control Variables real-time traffic state number of vehicles, queue length, flow emission and time spend emission and time spend number of vehicles, flow position and speed, estimated: number of arriving cars, accelerations; reduced: stops, departure times position, velocity Solution prediction, V2I communication in ad-hoc networks, called Vanet adaptive, centralised vs. distributed, interconnected intersections, inclusion of priority travellers central flow emission model, towards hierarchical central flow emission model, smoothening methods, gradient methods with resilient backpropagation hierarchical, signal split, coordination algorithm partially connected vehicles optimal linear quadratic, collision avoidance, inclusion of non-cooperative vehicles Table 17.4: Table of references on MPC - 1 Problem multi-junction traffic flow control intersection management congestion, emissions Traffic signal control intersection traffic control coordination of autonomous vehicles Field of Interest road traffic management urban traffic management urban traffic networks urban traffic networks Large-Scale Urban Road Networks intersection management intersection management Ref. [33] [48] [1] [23] [57] [20] [30] 236 L. E. Schneegans grading Variable average cost, green cycle length crossing time miles per gallon penetration rates green time, traffic flow, number of vehicles total vehicle delay, queue length occupancy, green time acceleration, velocity, path, cpu time congestion Control Variables flow mode, cycle length, vehicle capacity stop delay signal phase and timing information from trafc, stop time reduction total delay, flow, demand ratio green time, traffic flow, number of vehicles connected vehicle data, phase sequence, phase duration, penetration rate flow, total time spent, relative queue balance acceleration, velocity, path queue size, arrival rates, throughput Solution blocking, gradient-based, stochastic hybrid system coordination scheme without traffic lights, risk indicator for collision avoidance V2X communication dedicates from [20], control schemesdifferent for technical groups, departure sequence, trajectory linear quadratic, store and forward equation signal phase allocation, quadratic programming parallelised optimisation Back-pressure scheme Table 17.5: Table of references on MPC- 2 Problem multi-section autonomous driving urban road conditions mixed technologies of vehicles, autonomous driving signal split adaptive sign control signal control intersection automation signal control Field of Interest traffic light control intersection management intersection management isolated intersection control urban traffic networks single intersections large-scale urban traffic networks congestion in big cities urban road networks Ref. [19] [24] [21] [56] [37] [17] [2] [25] [28] 17 Survey on Urbantraffic Control and Management 237 exchange between the fields of electrical respective computer sciences with civil engineering and a leak of methods addressing strengths and weaknesses coming from both fields.

References

1. A. Jamshidnejad, M. Papageorgiou, I.P., Schutter, B.D.: a multi- objective model- predictive control approach dealing with congestion and emissions in urban trafc networks. Proceedings of the 1st ARTS ECR Conference, La Valletta, Malta (2015) 2. Aboudolas, K., et al.: A rolling-horizon quadratic-programming approach to the signal control problem in large-scale congested urban road networks. Transportation Research Part C: Emerging Technologies 18(5), 680–694 (2010) 3. Alam, M., et al.: Intelligent Transportation Systems. Studies in Systems, Decision and Control, vol. 52. Springer, Cham (2016) 4. Barfiled, W., Dingus, T.: Human factors in intelligent transportation Systems. Psychol- ogy Press (2014) 5. Barthelemy,´ J., Carletti, T.: A dynamic behavioural traffic assignment model with strategic agents. Transportation Research Part C: Emerging Technologies 85, 23–46 (dec 2017) 6. Bazzan, A.L.C., Klgl, F.: Reducing the Eects of the Braess Paradox with Information Manipulation (2005) 7. Bellifemine, F., Poggi, A.: Jade: a fipa2000 compliant agent development environment. In: AGENTS ’01 Proceedings of the fifth international conference on Autonomous agents. pp. 216–217 (2001) 8. Bui, K., Jung, J.: Cooperative game-theoretic approach to traffic flow optimization for multiple intersections. Computers & Electrical Engineering (oct 2017) 9. Camponogara, E., et al.: Distributed model predictive control. IEEE (2013) 10. Chen, B., Cheng, H.H.: A review of the applications of agent technology in traffic and transportation systems. IEEE Transactions on Intelligent Transportation Systems 11(2), 485–497 (jun 2010) 11. Chen, L., Englund, C.: Cooperative intersection management: A survey. IEEE Transac- tions on Intelligent Transportation Systems 17(2), 570–586 (feb 2016) 12. Christofides, P.D., Scattolini, R., de la Pena,˜ D.M., Liu, J.: Distributed model predictive control: A tutorial review and future research directions. Computers & Chemical Engineering 51, 21–41 (2013) 13. Davidsson, P., Henesey, L., Ramstedt, L., Trnquist, J., Wernstedt, F.: Agent-Based Ap- proaches to Transport Logistics (2005) 14. dpa: Rekord - 60 prozent der beschftigten pendeln (2016), http://www.spiegel. de/wirtschaft/service/pendlerstudie-60-prozent-muessen-zur-arbeit-pendeln-a- 1141519.html 15. Dresner, K., Stone, P.: Multiagent traffic management: A reservation-based intersection control mechanism (2004) 16. Fellendorf, M.: Vissim: A and microscopic simulation and tool to evaluate and actuated signal and control and including bus and priority (1994) 17. Feng, Y., et al.: A real-time adaptive signal control in a connected vehicle environment. Transportation Research Part C: Emerging Technologies 55, 460–473 (jun 2015) 238 L. E. Schneegans

18. Frejo, J., Camacho, E.: Global versus local MPC algorithms in freeway traffic control with ramp metering and variable speed limits. IEEE Transactions on Intelligent Trans- portation Systems 13(4), 1556–1565 (dec 2012) 19. Geng, Y., Cassandras, C.G.: Multi-intersection traffic light control with blocking. Dis- crete Event Dynamic Systems 25(1-2), 7–30 (nov 2013) 20. Guler, S.I., Menendez, M., Meier, L.: Using connected vehicle technology to improve the efficiency of intersections. Transportation Research Part C: Emerging Technologies 46, 121–131 (sep 2014) 21. HomChaudhuri, B., et al.: Fast model predictive control-based fuel efficient control strategy for a group of connected vehicles in urban road conditions. IEEE Transactions on Control Systems Technology 25(2), 760–767 (mar 2017) 22. Hoyer, R.: Potential effects of data interchange between vehicles and infrastructure at signalized intersections. In: 13 th World Conference on Transport Research (WCTR), Rio de Janeiro, Brazil, (2013) 23. Jamshidnejad, A., et al.: Sustainable model-predictive control in urban traffic networks: Efficient solution based on general smoothening methods. IEEE Transactions on Control Systems Technology 26(3), 813–827 (may 2018) 24. Kamal, M.A.S., et al.: A vehicle-intersection coordination scheme for smooth flows of traffic without using traffic lights. IEEE Transactions on Intelligent Transportation Sys- tems 16(3), 1136–1147 (jun 2015) 25. Katriniok, A., et al.: Distributed model predictive control for intersection automation using a parallelized optimization approach. IFAC-PapersOnLine 50(1), 5940–5946 (jul 2017) 26. Khamis, M.A., Gomaa, W.: Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework. Engineering Applications of Artificial Intelligence 29, 134–151 (2014) 27. Klgl, F., et al.: Application of Agent Technolgy in Traffic and Transportation (2005) 28. Le, T., et al.: Decentralized signal control for urban road networks. Transportation Re- search Part C: Emerging Technologies 58, 431–450 (sep 2015) 29. Maciejowski, J.M.: Predictive Control with Constraints. Prentice-Hall, Edinburgh (2002) 30. Makarem, L., Gillet, D.: Model predictive coordination of autonomous vehicles crossing intersections. In: Proceedings of the 16th International IEEE Annual Conference on TuD6 .3 Intelligent Transportation Systems (ITSC 2013), The Hague, The Netherlands, October 6-9, 2013. IEEE (2013) 31. Makarova, I., et al.: Increase of City Transport System Management Efficiency with Application of Modeling Methods and Data Intellectual Analysis (2016) 32. Mayne, D.: Model precitive control: Recent developements and future promise. Auto- matica (50), 2967–2986 (2014) 33. Nafi, N.S., et al.: A predictive road traffic management system based on vehicular ad-hoc network. Tech. Rep. paper. (2014) 34. Negenborn, R., Maestre, J.: Distributed model predictive control: An overview and roadmap of future research opportunities. IEEE Control Systems 34(4), 87–97 (aug 2014) 35. Negenborn, R.R., et al.: Multi-agent model predicitve control for transportation networks with continuous and discrete elements (2006) 36. Negenborn, R.R., et al.: Multi-agent model predictive control for transportation networks: Serial versus parallel schemes. Engineering Applications of Articial Intelligence 21 (2008) 353366 (2007) 17 Survey on Urbantraffic Control and Management 239

37. de Oliveira, L.B., Camponogara, E.: Multi-agent model predictive control of signaling split in urban traffic networks. Transportation Research Part C: Emerging Technologies 18(1), 120–139 (feb 2010) 38. Opfer, S., Witsch, A.: Cooperative behaviour of autonomous vehicles (2014) 39. Ossowski, S., et al.: Designing Multiagent Decision Support Systems for Trac Manage- ment (2005) 40. Otto, Th.; Hoyer, R.: Probe vehicle based estimation of parameters for adapting traffic signal control at single intersections. In: 17th World Congress on Intelligent Transport Systems and Services, Busan, South Korea (2010) 41. Pamua, T., Krl, A.: The Traffic Flow Prediction Using Bayesian and Neural Networks (2016) 42. Prothmann, H., Branke, J., Schmeck, H., Tomforde, S., Rochner, F., Hahner,¨ J., Muller-¨ Schloer, C.: Organic Traffic Light Control for Urban Road Networks. International Jour- nal of Autonomous and Adaptive Communications Systems 2(3), 203 – 225 (2009) 43. Prothmann, H., et al.: Organic control of traffic lights (general.) (2008) 44. Scattolini, R.: Architectures for distributed and hierarchical model predictive control – a review. Journal of Process Control 19(5), 723–731 (may 2009) 45. Schlingensiepen, J., et al.: Autonomic Transport Management SystemsEnabler for Smart Cities, Personalized Medicine, Participation and Industry Grid/Industry 4.0 (2016) 46. Statista GmBH: Anzahl der gemeldeten pkw in deutschland in den jahren 1960 bis 2018 (bestand in 1.000) (2018), https://de.statista.com/statistik/daten/studie/12131/umfrage/ pkw-bestand-in-deutschland/ 47. Sadkowski, A., Pamua, W.: Intelligent Transportation Systems Problems and Perspec- tives. Springer, Cham (2016) 48. Tams Tettamanti, I.V., Pni, T.: Mpc in urban trafic management. In: Model Predictive Control (2010) 49. Teodorovic, D.: Transport modeling by multi-agent systems: a swarm intelligence approach. Transportation Planning and Technology 26(4), 289–312 (aug 2003) 50. Tomforde, S., Prothmann, H., Branke, J., Hahner,¨ J., Muller-Schloer,¨ C., Schmeck, H.: Possibilities and Limitations of Decentralised Traffic Control Systems. In: 2010 IEEE World Congress on Computational Intelligence (IEEE WCCI 2010), held 18 Jul - 23 Jul 2010 in Narcelona, Spain. pp. 3298–3306. IEEE (2010) 51. Tomforde, S., Prothmann, H., Rochner, F., Branke, J., Hahner,¨ J., Muller-Schloer,¨ C., Schmeck, H.: Decentralised Progressive Signal Systems for Organic Traffic Control. In: Brueckner, S., Robertson, P., Bellur, U. (eds.) Proceedings of the 2nd IEEE International Conference on Self-Adaption and Self-Organization (SASO’08), held in Venice, Italy (October 20 - 24, 2008). pp. 413–422. IEEE (2008) 52. Touhbi, S., et al.: Adaptive traffic signal control : Exploring reward definition for reinforcement learning. Procedia Computer Science 109, 513–520 (2017) 53. Wang, F.Y.: Parallel control and management for intelligent transportation systems: Con- cepts, architectures, and applications. IEEE Transactions on Intelligent Transportation Systems 11(3), 630–638 (sep 2010) 54. Weisheit, T., Schaefer, M., Hoyer, R.: Support vector machines - a suitable approach for a prediction of switching times of traffic actuated signal controls. pp. 121–129 (2014) 55. Yamashita, T., et al.: Analysis of the Eect of Route Information Sharing on Reduction of Trac Congestion (2005) 56. Yang, K., et al.: Isolated intersection control for various levels of vehicle technology: Conventional, connected, and automated vehicles. Transportation Research Part C: Emerging Technologies 72, 109–129 (nov 2016) 240 L. E. Schneegans

57. Ye, B.L., et al.: A hierarchical model predictive control approach for signal splits optimization in large-scale urban road networks. IEEE Transactions on Intelligent Trans- portation Systems 17(8), 2182–2192 (aug 2016) 58. Zochowska, R., Karon, G.: ITS Services Packages as a Tool for Managing Trafﬁc Con- gestion in Cities 13 13

S. Tomforde | B. Sick (Eds.)

Organic Computing Doctoral Dissertation Colloquium 2018 B. Sick (Eds.) Organic Computing B. | S. Tomforde S. Tomforde

ISBN 978-3-7376-0696-7

9 783737 606967