High-Performance Computing Methods in Large-Scale Power System Simulation

Lukas Razik Institute for Automation of Complex Power Systems

Von der Fakultät für Elektrotechnik und Informationstechnik der Rheinisch-Westfälischen Technischen Hochschule Aachen zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften genehmigte Dissertation

vorgelegt von

Dipl.-Inform. Lukas Daniel Razik

aus Hindenburg

Berichter: Univ.-Prof. Antonello Monti, Ph. D. Univ.-Prof. Dr.-Ing. Andrea Benigni

Tag der mündlichen Prüfung: 8. Mai 2020

Diese Dissertation ist auf den Internetseiten der Universitätsbibliothek online verfügbar. Bibliographische Information der Deutschen Nationalbibliothek Die Deutsche Nationalbibliothek verzeichnet diese Publikation in der Deutschen Nationalbibliografie; detaillierte bibliografische Daten sind im Internet über http://dnb-nb.de abrufbar.

D 82 (Diss. RWTH Aachen University, 2020)

Herausgeber: Univ.-Prof. Dr.ir. Dr. h. c. Rik W. De Doncker Direktor E.ON Energy Research Center

Institute for Automation of Complex Power Systems (ACS) E.ON Energy Research Center Mathieustraße 10 52074 Aachen

E.ON Energy Research Center I 81. Ausgabe der Serie ACS I Automation of Complex Power Systems

Copyright Lukas Razik Alle Rechte, auch das des auszugsweisen Nachdrucks, der auszugsweisen oder vollständigen Wiedergabe, der Speicherung in Datenverarbeitungsanlagen und der Übersetzung, vorbehalten.

Printed in Germany

ISBN: 978-3-942789-80-6 1. Auflage 2020

Verlag: E.ON Energy Research Center, RWTH Aachen University Mathieustraße 10 52074 Aachen Internet: www.eonerc.rwth-aachen.de E-Mail: [email protected]

Zusammenfassung

In der seit 2009 geltenden Erneuerbare-Energien-Richtlinie der Europäi- schen Union haben sich die Mitgliedsstaaten darauf verständigt, dass der Anteil erneuerbarer Energien bis 2020 bei mindestens 20 % des Ener- gieverbrauchs liegen soll. Die damit einhergehende wachsende Zahl von erneuerbaren Energieerzeugern wie Photovoltaik- und Windkraftanlagen führt zu einer vermehrt dezentralen Stromerzeugung, die ein komplexeres Stromnetzmanagement erfordert. Um dennoch einen sicheren Netzbetrieb zu gewährleisten, ﬁndet ein Wandel von konventionellen Stromnetzen zu sogenannten Smart Grids statt, bei denen z. B. nicht nur Statusinformationen der Stromerzeuger son- dern auch der Verbraucher (z. B. Wärmepumpen und Elektrofahrzeuge) in das Netzmanagement einbezogen werden. Die Nutzung von Flexibilitäten auf Erzeugungs- und Nachfrageseite und der Einsatz von Energiespei- chern zur Erreichung einer stabilen und wirtschaftlichen Stromversorgung erfordert neue Lösungen für die Planung und den Betrieb von Smart Grids. Andernfalls können Veränderungen an den Systemen des öﬀentli- chen Energiesektors (Stromnetz, IKT-Infrastruktur, Energiemarkt usw.) zu unerwarteten Problemen und damit auch zu Stromausfällen führen. Computersimulationen können deswegen helfen, das Verhalten von Smart Grids bei Veränderungen abzuschätzen, ohne das Risiko negativer Folgen bei unausgereiften Lösungen oder Inkompatibilitäten einzugehen. Die wesentliche Zielsetzung der vorliegenden Dissertation ist die An- wendung und Analyse von Methoden des High-Performance Computings (HPC) und der Informatik zur Verbesserung von (Co-)Simulationssoftware elektrischer Energiesysteme, um komplexere Komponentenmodelle sowie größere Systemmodelle in angemessener Zeit simulieren zu können. Durch die zunehmende Automatisierung und Regelung in Smart Grids, die immer höheren Anforderungen an deren Flexibilität und die Notwendigkeit einer stärkeren Marktintegration der Verbraucher werden Stromnetzmodelle immer komplexer. Die Simulationen erfordern daher eine immer höhere

iii Leistungsfähigkeit der eingesetzten Rechnersysteme. Der Schwerpunkt der Arbeiten liegt deshalb auf der Verbesserung verschiedener Aspekte moderner und derzeit entwickelter Simulationslösungen. Dabei sollten jedoch keine neuen Simulationskonzepte oder -anwendungen entwickelt werden, die ein Hochleistungsrechnen auf Supercomputern oder großen Computerclustern erst erforderlich machen würden. Vielmehr werden in dieser Dissertation die Integrationen moderner direkter Löser für dünnbesetzte lineare Systeme in verschiedene Strom- netzsimulations-Backends und die anschließenden Analysen mithilfe von großskaligen Stromnetzmodellen vorgestellt. Darüber hinaus wird eine neue Methode zur automatischen grobgranularen Parallelisierung von Stromnetz-Systemmodellen auf Komponentenebene präsentiert. Neben solchen konkreten Anwendungen von HPC-Methoden auf Simulationsumge- bungen wird auch eine vergleichende Analyse verschiedener HPC-Ansätze zur Leistungssteigerung Python-basierter Software mithilfe von (Just-in- Time-)Kompilierern vorgestellt, da Python – in der Regel eine interpretierte Programmiersprache – im Bereich der Softwarenetwicklung im Energiesek- tor immer beliebter wird. Im Weiteren stellt die Dissertation die Integration einer HPC-Netzwerktechnologie auf Basis des offenen InfiniBand-Standards in ein Software-Framework vor, das für die Kopplung verschiedener Simu- lationsumgebungen zu einer Co-Simulation und für den Datenaustausch in Hardware-in-the-Loop (HiL) Aufbauten genutzt werden kann. Für die Verarbeitung von Energiesystemtopologien durch Simulations- umgebungen, auf denen die oben genannten HPC-Methoden angewendet wurden, ist die Unterstützung eines standardisierten Datenmodells not- wendig. Die Dissertation behandelt daher auch das Common Information Model (CIM), wie in IEC 61970 / 61968 standardisiert, welches für die Spezifikation von Datenmodellen zur Repräsentierung von Energiesystem- topologien verwendet werden kann. Zunächst wird ein gesamtheitliches Datenmodell vorgestellt, das für Co-Simulationen des Stromnetzes mit dem zugehörigen Kommunikationsnetz und dem Energiemarkt durch eine Erweiterung von CIM entwickelt wurde. Um eine nachhaltige Entwicklung von CIM-bezogenen Softwaretools zu erreichen, wird im Folgenden eine automatisierte (De-)Serializer-Generierung aus CIM-Spezifikationen vorgestellt. Die Deserialisierung von CIM-Dokumenten ist ein Schritt, der für die anschließend entwickelte Übersetzung von CIM-basierten Netztopologien in simulatorspezifische Systemmodelle genutzt wird, die ebenfalls in dieser Dissertation behandelt wird. Viele der vorgestellten Erkenntnisse und Ansätze können auch zur Ver- besserung anderer Software im Bereich der Elektrotechnik und darüber hinaus genutzt werden. Zudem wurden alle in der Dissertation vorgestell-

iv ten Ansätze in öﬀentlich zugänglichen Open-Source-Softwareprojekten implementiert.

Abstract

In the Renewables Directive of the European Union, in effect since 2009, the member states agreed that the share in renewable energy should be 20 % of the total energy by 2020. The concomitantly growing number of renewable energy producers such as photovoltaic systems and wind power plants leads to a more decentralized power generation. This results in a more complex power grid management. To ensure a secure power grid operation even so, there is a transformation from conventional power grids to so-called smart grids where, for instance, not only status information of power producers but also of consumers (e. g. heat pumps and electrical vehicles) is included in the power grid management. The utilization of flexibility on generation and demand side and the use of energy storage systems for achieving a stable and economic power supply requires new solutions for the planning and operation of smart grids. Otherwise, manipulations of the systems in the public energy sector (i. e. power grid, information and communications technology (ICT) infrastructure, energy market, etc.) can lead to unexpected problems such as power failures. Computer simulations therefore can help to estimate the behavior of smart grids on any changes without the risk of negative consequences in case of immature solutions or incompatibilities. The main objective of this dissertation is the application and analysis of high-performance computing (HPC) and computer science methods for improving power system (co-)simulation software to allow simulating more detailed models in a, for the particular use case, appropriate time. Through more automation and control in smart grids, the higher demand on flexibility, and the need of stronger market integration of consumers, the power system models become more and more complex. This requires an ever greater performance of the utilized computer systems. The focus was on the improvement of different aspects of state-of-the-art and currently developed simulation solutions. The intention was not to develop new

vii simulation concepts or applications that would make large-scale HPC on super-computers or large computer clusters necessary. The dissertation presents the integration of modern direct solvers for sparse linear systems in various power grid simulation back-ends and subsequent analyses with the aid of large-scale power grid models. Fur- thermore, a new method for an automatic coarse-grained parallelization of power grid system models at component level is shown. Besides such concrete applications of HPC methods on simulation environments, also a comparative analysis of various HPC approaches for performance improvement of Python based software with the aid of (just-in-time) compilers is presented, as Python – usually an interpreted programming language – becomes more popular in the area of power system related software. Moreover, the dissertation shows the integration of an HPC interconnect solution based on InfiniBand – an open standard – in a software framework for the coupling of different simulation environments to a co-simulation and for Hardware-in-the-Loop (HiL) setups. The support of a standardized data model for the processing of power system topologies by simulation environments, on which the aforemen- tioned HPC methods were applied, is necessary. Therefore, the dissertation concerns the Common Information Model (CIM) as, i. a., standardized by IEC 61970 / 61968, which can be used for the specification of data models representing power system topologies. At first, a holistic data model is introduced that was developed for co-simulations of the power grid with the associated communication network and the energy market by extending CIM. To achieve a sustainable development of CIM related software tools, an automated (de-)serializer generation from CIM spec- ifications is presented. The deserialization from CIM is a step needed for the subsequently developed template-based translation from CIM to simulator-specific system models which is also covered in this dissertation. Many presented findings and approaches can be used for improving further software from the area of electrical engineering and beyond that. Moreover, all presented approaches were implemented in open-source software projects, accessible by the public.

viii Acknowledgement

I would like to thank the following people: My doctoral supervisor, Prof. Antonello Monti, for the guidance and the support of my initiatives throughout my whole time as doctoral student at the Institute for Automation of Complex Power Systems, my second reviewer, Prof. Andrea Benigni, for the kind feedback to my dissertation manuscript, and Prof. Ferdinanda Ponci for the helpful feedback and support regarding my scientific publications. My colleagues Jan Dinkelbach, for reading the manuscript (especially the boring parts) and the great support during my way from a computer scientist to an engineer, Markus Mirz, for a great cooperation as well as the inclusion of my humble self in interesting additional projects and activities, Steffen Vogel for the assistance in software-technical matters, Simon Pickartz for the sophisticated LATEX template, and Stefan Dähling for proofreading the final version. All student researchers and students who participated in the research and development related to this dissertation. The Réseau de Transport d’Électricité co-workers Adrien Guironnet and Gautier Bureau for a successful and enjoyable cooperation.

Vor allem möchte ich meinen Eltern danken, die auf vieles verzichtet und es mir durch Ihre Unterstützung erst ermöglicht haben, diesen beruﬂichen Weg zu beschreiten. Zu guter Letzt danke auch Dir, mein Schatz, für Deine Unterstützung und Geduld während meiner Promotionszeit!

Aachen, May 2020 Lukas Daniel Razik

Contents

Acknowledgement viii

List of Publications xv

1 Introduction 1 1.1 Challenges in Smart Grids ...... 1 1.2 Large-Scale Multi-Domain Co-Simulation as a Solution .. 3 1.3 Contribution ...... 6 1.4 Outline ...... 11

2 Multi-Domain Co-Simulation 13 2.1 Fundamentals and Related Work ...... 14 2.1.1 Architecture and Topology Data Model ...... 14 2.1.2 Common Information Model ...... 15 2.1.3 Simulation of Smart Grids ...... 16 2.1.4 Classiﬁcation of Simulations ...... 16 2.2 Use Case ...... 17 2.3 Challenges ...... 18 2.4 Concept of the Co-Simulation Environment ...... 19 2.4.1 Holistic Topology Data Model ...... 19 2.4.2 Model Data Processing and Simulation Setup ... 22 2.4.3 Synchronization ...... 23 2.4.4 Co-Simulation Runtime Interaction ...... 24 2.5 Validation by Use Case ...... 26 2.6 Conclusion ...... 27

3 Automated De-/Serializer Generation 29 3.1 CIM Formalisms and Formats ...... 31 3.2 CIM++ Concept ...... 33

xi Contents

3.3 From CIM UML to Compilable C++ Code ...... 35 3.3.1 Gathering Generated CIM Sources ...... 37 3.3.2 Refactoring Generated CIM Sources ...... 38 3.3.3 Primitive CIM Data Types ...... 40 3.4 Automated CIM (De-)Serializer Generation ...... 41 3.4.1 The Common Base Class ...... 41 3.4.2 Integrating an XML Parser ...... 42 3.4.3 Unmarshalling ...... 43 3.4.4 Unmarshalling Code Generator ...... 46 3.4.5 Marshalling ...... 49 3.5 libcimpp Implementation ...... 50 3.6 Evaluation ...... 50 3.7 Conclusion and Outlook ...... 51

4 From CIM to Simulator-Speciﬁc System Models 55 4.1 CIMverter Fundamentals ...... 57 4.1.1 Modelica ...... 57 4.1.2 Template Engine ...... 59 4.2 CIMverter Concept ...... 59 4.3 CIMverter Implementation ...... 62 4.3.1 Mapping from CIM to Modelica ...... 63 4.3.2 CIM Object Handler ...... 64 4.4 Modelica Workshop Implementation ...... 65 4.4.1 Base Class of the Modelica Workshop ...... 66 4.4.2 CIM to Modelica Object Mapping ...... 66 4.4.3 Component Connections ...... 67 4.5 Evaluation ...... 68 4.6 Conclusion and Outlook ...... 70

5 Modern LU Decompositions in Power Grid Simulation 75 5.1 LU Decompositions in Power Grid Simulation ...... 76 5.1.1 From DAEs to LU Decompositions ...... 76 5.1.2 LU Decompositions for Linear System Solving .. 78 5.1.3 KLU, NICSLU, GLU, and Basker by Comparison . 80 5.2 Analysis of Modern LU Decompositions for Electrical Circuits 83 5.2.1 Analysis on Benchmark Matrices from Large-Scale Grids ...... 84 5.2.2 Analysis on Power Grid Simulations ...... 92 5.3 Conclusion and Outlook ...... 95

xii Contents

6 Exploiting Parallelism in Power Grid Simulation 97 6.1 Parallelism in Simulation Models ...... 98 6.1.1 Task Scheduling ...... 100 6.1.2 Task Parallelization in DPsim ...... 106 6.1.3 System Decoupling ...... 110 6.2 Analysis of Task Parallelization in DPsim ...... 111 6.2.1 Use Cases ...... 112 6.2.2 Schedulers ...... 113 6.2.3 System Decoupling ...... 117 6.2.4 Compiler Environments ...... 122 6.3 Conclusion and Outlook ...... 124

7 HPC Python Internals and Beneﬁts 127 7.1 HPC Python Fundamentals ...... 129 7.1.1 Classical Python ...... 130 7.1.2 PyPy ...... 136 7.1.3 Numba ...... 139 7.1.4 Cython ...... 143 7.2 Benchmarking Methodology ...... 147 7.3 Comparative Analysis ...... 150 7.4 Conclusion and Outlook ...... 155

8 HPC Network Communication for HiL and RT Co-Simulation 157 8.1 VILLAS Fundamentals ...... 158 8.2 InfiniBand Fundamentals ...... 159 8.2.1 InfiniBand Architecture ...... 161 8.2.2 OpenFabrics Software Stack ...... 165 8.3 Concept of InfiniBand Support in VILLAS ...... 167 8.3.1 VILLASnode Basics ...... 167 8.3.2 Original Read and Write Interface ...... 167 8.3.3 Requirements on InfiniBand Node-Type Interface . 170 8.3.4 Memory Management of InfiniBand Node-Type .. 171 8.3.5 States of InfiniBand Node-Type ...... 172 8.3.6 Implementation of InfiniBand Node-Type ..... 173 8.4 Analysis of the InfiniBand Support in VILLAS ...... 175 8.4.1 Service Types of InfiniBand Node-Type ...... 178 8.4.2 InfiniBand vs. Zero-Latency Node-Type ...... 181 8.4.3 InfiniBand vs. Existing Server-Server Node-Types 182 8.5 Conclusion and Outlook ...... 183

9 Conclusion 185 9.1 Summary and Discussion ...... 185

xiii Contents

9.2 Outlook ...... 189

A Code Listings 193 A.1 Exploiting Parallelism in Power Grid Simulation ..... 193

B Python Environment Measurements 195 B.1 Execution Times ...... 195 B.2 Memory Space Consumption ...... 197

List of Acronyms 201

Glossary 207

List of Figures 209

List of Tables 213

Bibliography 215

xiv List of Publications

Journal Articles

[DRM20] S. Dähling, L. Razik, and A. Monti. “OWL2Go: Auto-generation of Go data models for OWL ontologies with integrated serialization and deserialization functionality”. In: To appear in SoftwareX (2020). [Raz+19b] L. Razik, N. Berr, S. Khayyam, F. Ponci, and A. Monti. “REM-S-–Railway Energy Management in Real Rail Opera- tion”. In: IEEE Transactions on Vehicular Technology 68.2 (Feb. 2019), pp. 1266–1277. doi: 10.1109/TVT.2018.2885007. [Kha+18] S. Khayyamim, N. Berr, L. Razik, M. Fleck, F. Ponci, and A. Monti. “Railway System Energy Management Optimiza- tion Demonstrated at Oﬄine and Online Case Studies”. In: IEEE Transactions on Intelligent Transportation Systems 19.11 (Nov. 2018), pp. 3570–3583. issn: 1524-9050. doi: 10. 1109/TITS.2018.2855748. [Mir+18] M. Mirz, L. Razik, J. Dinkelbach, H. A. Tokel, G. Alirezaei, R. Mathar, and A. Monti. “A Cosimulation Architecture for Power System, Communication, and Market in the Smart Grid”. In: Hindawi Complexity 2018 (Feb. 2018). doi: 10. 1155/2018/7154031. [Raz+18a] L. Razik, M. Mirz, D. Knibbe, S. Lankes, and A. Monti. “Auto- mated deserializer generation from CIM ontologies: CIM++— an easy-to-use and automated adaptable open-source library for object deserialization in C++ from documents based on user-speciﬁed UML models following the Common Infor- mation Model (CIM) standards for the energy sector”. In: Computer Science - Research and Development 33.1 (Feb.

xv Contents

2018), pp. 93–103. issn: 1865-2042. doi: 10.1007/s00450- 017-0350-y. [Raz+18b] L. Razik, J. Dinkelbach, M. Mirz, and A. Monti. “CIMverter— a template-based ﬂexibly extensible open-source converter from CIM to Modelica”. In: Energy Informatics 1.1 (Oct. 2018), p. 47. issn: 2520-8942. doi: 10.1186/s42162-018- 0031-5. [Gre+16] F. Gremse, A. Höfter, L. Razik, F. Kiessling, and U. Nau- mann. “GPU-accelerated adjoint algorithmic diﬀerentiation”. In: Computer Physics Communications 200 (2016), pp. 300– 311. issn: 0010-4655. doi: 10.1016/j.cpc.2015.10.027. [Fin+09b] R. Finocchiaro, L. Razik, S. Lankes, and T. Bemmerl. “Low- Latency Linux Drivers for Ethernet over High-Speed Net- works”. In: IAENG International Journal of Computer Sci- ence 36.4 (2009).

Book Chapters

[Fin+10] R. Finocchiaro, L. Razik, S. Lankes, and T. Bemmerl. “Trans- parent Integration of a Low-Latency Linux Driver for Dolphin SCI and DX”. In: Electronic Engineering and Computing Tech- nology. Ed. by S.-I. Ao and L. Gelman. Dordrecht: Springer Netherlands, 2010, pp. 539–549. isbn: 978-90-481-8776-8. doi: 10.1007/978-90-481-8776-8_46.

Conference Articles

[Raz+19a] L. Razik, L. Schumacher, A. Monti, A. Guironnet, and G. Bureau. “A comparative analysis of LU decomposition methods for power system simulations”. In: 2019 IEEE Milan PowerTech. June 2019, pp. 1–6. [Vog+17] S. Vogel, M. Mirz, L. Razik, and A. Monti. “An Open Solution for Next-generation Real-time Power System Simulation”. In: 2017 IEEE Conference on Energy Internet and Energy System Integration (EI2). Nov. 2017, pp. 1–6. doi: 10.1109/EI2. 2017.8245739.

xvi Conference Articles

[Pic+16] S. Pickartz, N. Eiling, S. Lankes, L. Razik, and A. Monti. “Mi- grating LinuX Containers Using CRIU”. In: High Performance Computing. Ed. by M. Taufer, B. Mohr, and J. M. Kunkel. Cham: Springer International Publishing, 2016, pp. 674–684. isbn: 978-3-319-46079-6. [Var+11] E. Varnik, L. Razik, V. Mosenkis, and U. Naumann. “Fast Conservative Estimation of Hessian Sparsity”. In: Fifth SIAM Workshop on Combinatorial Scientiﬁc Computing, May 19–21, 2011, Darmstadt, Germany. May 2011, pp. 18–21. [Fin+09a] R. Finocchiaro, L. Razik, S. Lankes, and T. Bemmerl. “ETH- OM, an Ethernet over SCI and DX Driver for Linux”. In: Proceedings of 2009 International Conference of Parallel and Distributed Computing (ICPDC 2009), London, UK. 2009. [Fin+08] R. Finocchiaro, L. Razik, S. Lankes, and T. Bemmerl. “ETHOS, a generic Ethernet over Sockets Driver for Linux”. In: Proceed- ings of the 20th IASTED International Conference. Vol. 631. 017. 2008, p. 239.

xvii

Introduction

In 1993 there were announcements in German newspapers saying that sun, water, and wind will also in long-range not cover more than 4 percent of the electricity demand [Büt16]. As early as 2007 their share on the electricity supply amounted to 14.2 percent. This was also the year when the first official definition of smart grid was provided by the Energy Independence and Security Act of 2007 [Con07] which was approved by the US Congress in January 2007. Meanwhile, the term smart grid is used worldwide for research, development, and investment programs with regard to technology innovations and the expansion of power grids. The principle approaches for the transformation of conventional power grids to smart grids were developed in the experts team Advisory Council of the Technology Platform for Europeans in the years from 2005 to 2008 to establish a conceptual base for a secure grid integration of significant electrical generation capacities on the basis of renewable, mostly volatile and weather-dependent energy sources.

1.1 Challenges in Smart Grids

Smart grids particularly require an improved coordination of grid operation and grid user behavior with the aid of information and communications technology (ICT) and with the objective to ensure the sustainability of economical, reliable, secure as well as ecofriendly power supply in the environment of increased energy eﬃciency and decreased greenhouse gas emissions. For instance, Smart Distribution plays a major role in the area

1 Chapter 1 Introduction of smart grids. It can be divided into three pillars with the following challenges [BS14a; BS14b]: 1. Automation and remote control of local distribution grids: e. g. voltage control at distribution level (traditional as well as including the grid users), possibilities of power ﬂow control, accelerated fault location and resumption of normal grid operation, as well as enhanced protection concepts;

2. Flexibility by virtual power plants (VPPs): i. e. demand side management and beneﬁts of VPPs in a perspectival market organization;

3. Smart Metering and market integration of consumers: i. e. dynamic tariffs, demand side response, and electromobility. Since the efficiency of proper solutions can be improved by collecting and analyzing of information (i. e. data), a research field called Energy Informatics has been established around 2012 with conferences such as the DACH+ Conference on Energy Informatics and the ACM e-Energy Conference. Furthermore, Centers for Energy Informatics for example at the University of Southern Denmark and the University of Southern California were founded to address ICT challenges of smart grids, e. g. with the help of artificial intelligence and machine learning approaches. However, new approaches and international standards in the area of ICT are not sufficient. Besides regulatory (i. e. legal) aspects, the introduction of new market rules is also necessary. A successful realization of the European goals for global warming gases reduction, the increase of energy efficiency as well as the continuously rising use of renewable energy sources require a harmonized design of the interrelations between all participants in the process of electrical power supply [BS14a; BS14b]. Both, the proponents as well as the opponents of renewable energies agree: the contribution of 37.8 percent from renewable energy sources to gross electricity consumption in Germany [Umw19] can be significantly increased in the long term only with a utilization of flexibility (on generation and demand side) and the use of energy storage systems. Since modifications of the subsystems involved in the energy sector (i. e. power grid, ICT infrastructure, energy market, etc.) involve the risk of technical and economical faults such as destabilizations, major changes should not be made without an accurate analysis of possible effects on the power system. Computer simulations can help to estimate the behavior of systems on their modifications with the aid of mathematical models to avoid negative consequences that could occur in real systems. In the

2 1.2 Large-Scale Multi-Domain Co-Simulation as a Solution following, diﬀerent kinds of power system simulation are introduced and it is motivated why large-scale multi-domain co-simulation is a solution for the here presented three pillars of challenges in smart grids.

1.2 Large-Scale Multi-Domain Co-Simulation as a Solution

There are different types of (co-)simulations depending on their goals. Depending on the considered aspects, simulation types can be classified by mathematical models with, e. g., pure algebraic equations for steady-state observations or ordinary differential equations (ODEs) for dynamic observations; simulation time which, e. g., can be continuous for “floating” physical processes or discrete for events that occur at particular points in time, marking a change of the system’s state; orchestration which, e. g., can be hybrid, when multiple system models from different domains are simulated by the same solver, or a co- simulation, when multiple system models are computed by different simulation solvers which are coupled (i. e. exchanging information during the simulation) [Sch+15]. Obviously, this list is not complete. The next sections, however, shall provide a more general overview of simulation types with some of their goals to motivate the contribution of this work. First, it shall be differentiated between online and offline simulations.

Online Simulation Online simulations are, e. g., performed for steady-state security assessment (SSA) and dynamic security assessment (DSA). In case of SSA, power ﬂow simulations of a sequence of (n-1)-states are needed to examine the abid- ance of the principle that, under the predicted maximum transmission and supply responsibilities, the grid security is ensured even when a component such as a transformer or a line becomes unplanned inoperative. The DSA is based on dynamic simulation which supplements the steady-state grid security calculations with calculations of power plant dynamics in case of close short circuits and grid equipment outage. These dynamic stability calculations can be very time-consuming. In Germany this means that for a timely availability of DSA results, to be used as decision aid for the dispatcher, all (n-1)-scenarios should be available within 5 minutes.

3 Chapter 1 Introduction

This means that around 100 dynamic stability calculations must be accomplished within this time frame. Such a real-time (RT) requirement is very challenging and hence requires an intelligent management of the calculation cases as well as low simulation execution times [BS14a; BS14b].

Oﬄine Simulation

Oﬄine simulations (steady-state, electromagnetic transient simulation (EMT), etc.) are performed, e. g., for grid expansion planning, maintenance planning, commissioning of new operating equipment, and so forth. As oﬄine simulations are not performed simultaneously to grid operation, they do not have any RT requirements. Nevertheless, low simulation execution times are important to obtain simulation results in acceptable time frames when many use cases or scenarios (e. g. the same power grid with various switching events, changing its topology during simulation) have to be simulated or in case of simulation models with thousands of nodes.

Large-Scale Simulation

Such simulations with several thousand of nodes, called large-scale, become important when simulation environments shall be applicable also on real-world scenarios rather than for lab experiments only. Though there are commercial simulation tools which allow large-scale power grid simulations for certain use cases, they have a significant disadvantage: they are closed source and thus changes of existing models (i. e. component models) or the solvers are often not possible. However, further development of models is an essential concern of scientific research to adapt them for future applications in smart grids as the lack of inertia in power grids through a decreasing ratio of big power generators and more distributed energy resources (DERs) can lead to frequency instabilities that cannot be simulated by conventional models. Hence, at the Institute for Automation of Complex Power Systems (ACS) new methods and concepts are implemented into open-source software which can be used and improved by everyone. Here, it should be noted that not only publicly funded scientific facilities can benefit from open-source simulation software but also economic enterprises, of which some increasingly count on open-source alternatives instead of closed source products. For instance, RTE-France, the French transmission system operator (TSO), also develops open-source simulation environments such as Dynaωo [Gui+18]. But, as in case of commercial software, compliance with international standards of associations such as CIGRE, IEC, IEEE, and VDE are crucial for the comparability

4 1.2 Large-Scale Multi-Domain Co-Simulation as a Solution of solution approaches, study results, and applicability in existing system environments.

Co-Simulation

Especially in case of co-simulation – a deﬁnition is given by [Sch+15]– where multiple simulators are coupled together, standardized data models for information exchange between them are usually necessary. There are single-domain and multi-domain co-simulations. Single domain co-simulations can be conducted with and without RT requirements. Without RT requirements, co-simulations can be useful if the involved simulators have complementary features but there is no need for a synchronization of the simulation time with the real time (i. e. the co-simulation time can run slower or faster than the wall clock). Particularly in the power grids domain, RT requirements can come into play, e. g. with (power) Hardware-in-the-Loop (HiL), Control-in-the-Loop (CiL), and Software-in-the-Loop (SiL) use cases where a solution (i. e. an embedded system such as a control device) has to be connected to a simulated environment to verify its correct functioning within a real environment. A special case hereof is the geographically distributed real- time simulation (GD-RTS) which is based on the concept of a virtual interconnection of laboratories in real time [Mon+18]. In this concept, a monolithic simulation model is partitioned into subsystem portions that are simulated concurrently on multiple digital real-time simulators (DRTSs). As a result, comprehensive and large-scale real-world scenarios can be simulated for the validation of the interoperability of novel hardware and software solutions with the existing power grid and without the need for any arbitrary in-the-Loop setups to be located at the same facility. Multi-domain co-simulation in the following denotes a coupling of one or more power grid simulators with other simulators of diﬀerent domains such as ICT, market, weather, and so on, for the obtainment of a holistic view of the power system. Therefore, the term power system in this work does not stand for power grid only but for the power grid together with any associated system such as the ICT infrastructure and the energy market in a holistic view. This is the key for an extensive analysis and understanding of smart grids as depicted in [BS14a; BS14b]. In the previous sections, the use of large-scale single- and multi-domain co-simulation as a solution for the analysis and development of smart grids with a continually growing share of renewable energy sources is motivated. The merits through the application of simulations during power system operation as well as for the planning of power systems as conducted since decades is undisputed.

5 Chapter 1 Introduction

Due to the three main challenges arising through the transition to smart grids (see Sect. 1.1) – more automation and control in local distribution grids (e. g. because of the needed digitalization), the higher demand on ﬂexibility (e. g. by demand side management), and the need of stronger market integration of consumers – the power system models become more and more complex. This requires an ever greater performance of the utilized computer systems.

1.3 Contribution

The main objective of this dissertation is the application of high-performance computing (HPC) methods in the area of Energy Informatics and their analysis for improving power system (co-)simulation software to allow simulating more complex component models as well as larger system models in an appropriate time. While in the past processor performance increased continuously with increasing CPU clock rates, since around 2005 this is not the case anymore because of the power wall [Bos11]. From then on, computer performance was increased by a growing numbers of cores per processor and accelerators such as, e. g., graphic processing units (GPUs) and Intel Xeon Phi adapters. Nowadays, the power draw is not a problem of central processing units (CPUs) only but also of whole supercomputers. Therefore, while the trend to more parallelism continues, HPC system designers are more and more turning to hardware architectures respec- tive accelerators with high power efficiency (usually measured by FLOPS per watt) like GPUs, Advanced RISC Machines (ARM) processor based systems or field-programmable gate array (FPGA) accelerators [Gag+19]. As in case of multi-core and manycore systems with special instructions sets for performance improvements (e. g. vector instructions), the software nowadays must be adapted continuously to make use of such new hardware features and accelerators. Under these circumstances, the focus is on the improvement of different aspects on state-of-the-art and currently developed simulation solutions in academic area as well as enterprises. Thus, the intention was not to develop new simulation concepts or applications that would make large-scale HPC on supercomputers or large computer clusters necessary. However, especially computer and network hardware of modern commodity clusters is in focus of the contribution. Figure 1.1 shows the real-world challenge of an improved coordination of smart grid operation and grid user behavior. This is addressed by a solution based on an appropriate and therefore increasingly complex modeling as well as (co-)simulation for smart grid planning and operation. The three major aspects of the solution, the contribution of this work

6 1.3 Contribution

Transition of conventional power grids to smart grids

Challenge: Smart grids require improved coordination of grid operation and grid user behavior

Solution: Appropriate and more complex modeling and (co-)simulation for smart grid planning and operation

Information Modeling Simulation exchange

Chapter 3: Chapter 5: Chapter 7: Automated Modern LU HPC Python (de-)serializer decompositions internals and generation from in power grid beneﬁts CIM UML simulation

Chapter 2: Chapter 4: Chapter 6: Chapter 8: Multi-domain co- From CIM-based Exploiting HPC network simulation with a topologies to parallelism in communication holistic CIM- simulator-speciﬁc power grid for HiL and RT based topology system models simulation co-simulation data model

High-performance computing and energy informatics

Figure 1.1: Contribution overview of this work

7 Chapter 1 Introduction refers to, are modeling, simulation, and information exchange. Arrows from bottom to the top illustrate the contribution of this work to these major aspects of large-scale power system (co-)simulation. On the one hand, mathematical component models become more complex, for example because of an increasing use of power electronics, on the other hand, the complexity of system models increases for example because of new electrical equipment and facilities, in case of smart grids ever more often with connections to other domains such as ICT, weather, mechanics, energy market, etc. which also need to be simulated. There- fore, a contribution of this thesis is the presentation of a multi-domain co-simulation architecture with a holistic (i. e. multi-domain) topology data model which is based on the Common Information Model (CIM) as standardized by IEC 61970 / 61968 / 62325, describing terms in the energy sector and relations between them. CIM plays an important role as it belongs to the IEC core semantic standards for smart grids [LE]. CIM makes use of the Unified Modeling Language (UML) which is state-of-the-art in computer science for the specification of classes and their relationships in object-oriented software design. This thesis therefore contributes with the concept of an automated (de-)serializer generation from a specification based on UML. Among others, the automated code generation process implements a CIM data model in C++ according to the given UML specification. It can be applied whenever the UML specification changes between its versions what usually happens a couple of times per year. This avoids manual changes in a code base with currently around one thousand classes and many relations between each other which would be very time-consuming and error-prone. The resulting (de-)serializer allows reading in CIM documents in C++, according to the CIM-based data model, modifying the data in the main memory, and writing the data into CIM documents. Due to CIM’s fine granularity over several abstraction levels, a component (e. g. power transformer) consists of many CIM objects. This is a reason why a mapping from CIM to a simulator-specific system model is intricate. However, when a mapping to a system model of a certain simulator is achieved, the mapping often can also be used for system models of different simulators. Therefore, a template-based mapping from CIM to system models is proposed. The templates allow a specification on how model parameters from a CIM document have to be written into the simulator-specific system model target format. The advantage of templates is that if the system model format is written in a given language (e. g. Modelica), the templates are written in the same language with place holders for the data from a CIM object to be mapped. Therefore, the user

8 1.3 Contribution does not need to learn another language for specifying the system model format. Simulation environments make use of various approaches for the transition from a system model to a system of mathematical equations, which is solved by the simulation solver. In case of power grid simulations, for instance, the resistive companion approach in combination with the New- ton(-Raphson) method can be applied which results in a linear system of equations (LSE), for each time step and Newton iteration. In another approach, all component models can be combined into a differential-algebraic system of equations (DAE) which is then passed to a DAE solver which finally linearizes it to LSEs as well. For power grid simulations, LSEs typically are very sparse (i. e. the fraction of non-zero elements in the matrix is typically much less than 1 ) and therefore require appropriate LSE solvers. The contribution in thish work is a comparative analysis of several modern LU decompositions for the solution of sparse LSEs coming from power grids against KLU 1 which is a well-established LU decomposition for electric circuits and therefore taken as the reference. The LU decompositions concerned are called modern as they are developed especially for current multi-core or massively parallel computer architectures. The comparison is based on benchmark matrices that arose during power grid simulation and simulations performed by existing simulation environments in which the most promising LU decompositions were integrated. There are various methods of expressing parallelism in power system simulation. On the one hand, the processing within a simulation solver can be parallelized for instance with the aid of a certain parallel programming paradigm in the solver’s programming language (e. g. with parallel constructs using OpenMP in C++ [Ope19b]). Similarly to that approach, on the other hand, parallelism in a system model can also be expressed with the aid of a formalism for parallel structures in the model (e. g. with parallel constructs in the modeling language ParModelica [Geb+12]). Besides such an explicit expression of parallelism in the solver or model, it is also possible to extract parallelism, e. g., from mathematical models at equation level which is a variant of already existing automatic fine-grained parallelization of mathematical models. The contribution of this work is, however, the introduction of an automatic exploitation of parallelism in system models at component level and therefore called an automatic coarse-grained parallelization of mathematical models. For this coarse- grained parallelization of mathematical models, parallel task schedulings are introduced. Accordingly, various task schedulers allow the parallel

1The “K” in KLU stands for “Clark Kent” which is the bourgeois identity behind the ﬁctional superhero Superman. This is an allusion to SuperLU, which is a well-known LU decomposition for sparse linear systems [DP10].

9 Chapter 1 Introduction processing of component models related tasks within one simulation step. An analysis of the whole implementation shows the execution time speedups with respect to different scheduling methods and other modeling and software technical aspects. Power system simulation requires not only simulation itself but also data processing before the simulation (e. g. load and generation profiles), during the simulation (e. g. data exchanged between simulators), and after the simulation (e. g. simulation results). Since Python, as a modern and relatively easy-to-learn script language, is enjoying ever growing popularity under programming beginners, many power engineers program diverse parts of software projects in the area of power system simulation in Python. Especially the pre- and postprocessing of simulation data is performed in Python, while the simulation cores are often programmed in other programming languages such as C++. Sometimes execution times of (usually interpreted) Python applications are too long for given use cases and there is not enough time or a lack of know-how to port the Python application to a more runtime-efficient language such as, e. g., C++. Admittedly, there are Python modules, just-in-time (JIT) compilers, and Python language extensions which allow improving the runtime efficiency of Python programs but their internals and benefits are rather unknown. The contribution of this work is therefore an overview and comparative analysis of the most popular approaches for the performance improvement of Python, not necessarily with the aid of parallelization (e. g. multithreading). Co-simulations as well as HiL setups require an information exchange between simulators as well as between devices and simulators. Especially in case of RT applications, short latencies in information exchange can be crucial. To reduce latencies, HPC interconnects, in contrast to commonly used interconnects, provide connection modes in which data is directly transmitted to or read from the main memory of a remote server without involving the operating system or a process running on a remote server as it is usually the case. Therefore, a contribution of this thesis is the presentation of InfiniBand (IB), a widely-used HPC network communication standard, and its integration in a state-of-the-art software framework that can, for instance, be freely used for hard RT coupling of devices with simulators in case of HiL setups as well as for the coupling of simulators in case of hard RT co-simulations with very low latencies. All the contributed approaches were implemented or integrated in existing or new open-source software projects which can be used and investi- gated. Moreover, the concepts and analyses introduced in this work for an improved modeling, simulation, and information exchange shall support other researchers, developers and users of (co-)simulation software.

10 1.4 Outline

1.4 Outline

Chapter 2 shows the benefits of multi-domain co-simulation and introduces an appropriate co-simulation environment for the three smart grid domains power grid, communication network, and energy market, developed in the research project SINERGIEN. This SINERGIEN environment is the start for several approaches, concepts, and analyses which are presented in the following chapters. The usage of UML for the specification of CIM allows extending it to a holistic topology data model that is used for the SINERGIEN co-simulation environment with simulators for the three mentioned domains. Chapter 3 presents the automated (de-)serializer generation from a specification based on UML. The automated deserializer generation is implemented in the CIM++ software project which can map CIM, as specified by UML, to a C++ code base, also implementing the holistic CIM-based data model. The thus created open-source software library allows reading and writing arbitrary CIM-based documents in C++. Chapter 4 shows the approach on how CIM-based documents for power grid topology representation can be translated into simulator-specific system models with the aid of template documents. In the SINERGIEN environment this became necessary for the power grid simulator based on Modelica to run simulations of power grid topologies stored in CIM-based documents, as CIM is used more and more by distribution system operators (DSOs) and TSOs. The translation from CIM to a simulator-specific model was implemented in the open-source software CIMverter. It uses template documents making it possible to modify the simulator-specific system models to be outputted in case the input format of the target simulator is changing, e. g. because of a newer version which allows to set more parameters or new component models to be included in a system model. This allows a flexible adaption of the translation from CIM to a supported simulator-specific model without a recompilation of CIMverter which is also shown in this chapter. Chapter 5 outlines the comparative analysis of several modern LU decompositions for sparse linear systems. In the first part of the analysis they are compared by different benchmark matrices arising from simulations of large-scale power grids. This analysis was a help for deciding which LU decomposition is worth to be integrated into existing simulation environments. In the second part, the most promising modern decomposition (after its integration) is compared with the reference decomposition by simulations with both a fixed time step and a variable time step solver. Therefore, these LU decompositions were i. a. integrated into the DAE

11 Chapter 1 Introduction solver used by the open-source simulation environments OpenModelica and Dynaωo. Chapter 6 presents the approach for exploiting parallelism in power grid simulation from the newly introduced type of approaches described as automatic coarse-grained parallelization of mathematical models for a higher performance through thereby enabled parallel computations in power system simulators. This approach is applied on a newly developed open-source power grid simulator called DPsim. At first, the implemented parallelization approach is categorized into the existing parallelism categories of simulation models. Moreover, an overview of formally defined scheduling methods for the parallel processing of data independent tasks is provided. It follows a performance analysis of the implemented task parallelization methods. Chapter 7 provides an overview of the internals of HPC approaches to improve the runtime of Python applications and an comparative analysis of these approaches. The comparative analysis is based on various benchmark algorithms of different algorithm classes that were programmed in Python and in C++, as an efficient reference. This comparative analysis can help Python programmers to chose the right approach for increasing the performance of Python applications with or without multithreading, with threads that are executed really in parallel which is not always the case in Python as will be explained, too. Chapter 8 presents the integration of a HPC network communication into HiL and RT co-simulation. The HPC interconnect solution chosen for the integration in the open-source VILLASframework, that can be utilized for the setup of HiL simulations and (even hard) RT coupling of DRTSs, is based on IB. IB was chosen as it is an open standard that is implemented by various manufacturers. The integration of IB is also compared with other communication methods provided by the VILLASframework. Chapter 9 concludes the dissertation, providing a summary and discussion on all topics of this work. Moreover, it gives an overview of future work that can be conducted for an improvement of the introduced concepts as well as their analyses and implementations.

12 2

Multi-Domain Co-Simulation

More and more distributed energy resources (DERs) at distribution level cause bidirectional power ﬂows between distribution and transmission level which require changes in the related information and communications technology (ICT) and energy market mechanisms. Therewith associated extension of the measurement devices in lower voltage layers, for instance, require appropriate communication network capabilities for meeting the requirements on the exchange of measurement data between the measurement devices and all involved entities such as control centers and substations. Therefore, electrical grids and the belonging communication networks should be planned holistically to take the interactions between both domains into account [Li+14]. Apart from that, new energy market models are developed for customers (i. e. prosumers) to empower them to a more active role in the exchange of energy with the grid [WH16] in a way that their behavior will be considered in grid operation [EFF15] and possibly vice versa. Given these facts, it is reasonable to include also the energy market simulation in the planning to get a holistic picture of future grids. An integration of energy market mechanisms, the communication network, and power grid hamper future studies on power grids due to a lack of established modeling approaches which encompass the three domains and there are only few tools which enable a joint simulation. In this chapter a comprehensive data model is presented together with a co-simulation architecture based on it. Both enable an investigation of dynamic interactions between power grid, communication network and market. Such interactions can be technical constraints of the grid which require actions

13 Chapter 2 Multi-Domain Co-Simulation on market side as well as communication failures which affect the communication between grid and market or market decisions that change the behavior of a generation unit or energy prosumers connected to the grid. For this purpose, a data model based on the Common Information Model (CIM) as standardized in IEC 61970/61968/62235 was created to be able to describe an entire smart grid topology with components and actors from all three domains. This data model is called SINERGIEN_CIM as it resulted in the research project SINERGIEN. It allows the storage of the whole network topology with components from all three domains in a single well-defined data model, hiding some complexity of the simulation from the users. SINERGIEN_CIM-based topology descriptions are being processed by the co-simulation architectures as presented in [Mir+18]. Some parts of the SINERGIEN co-simulation architecture will be addressed in the following as they are relevant for the research and development that is presented in the following chapters of this dissertation. After a section on the related work and another one on various use cases for multi-domain simulation, the challenges for the realization of the implemented SINERGIEN co-simulation environment are discussed. It follows a section about the concept and a further one on its validation by a use case. The chapter is concluded with final remarks in its last section. The work in this chapter has been partially presented in [Mir+18]1.

2.1 Fundamentals and Related Work

2.1.1 Architecture and Topology Data Model

A major formal modeling method for future intelligent power grids is given by the Smart Grid Architecture Model (SGAM) [Sma12]. Therefore, the SGAM framework provides five levels: for physical components in the network (component layer), protocols for information exchange between services or systems (communication layer), data models which define the rules for data structures (information layer), functions and services (function layer), as well as business and market models (business layer). Furthermore, the model divides all two-dimensional layers into the domain dimension from generation over transmission, and so forth to customer premises and in the zones dimension from process over field, and so on, to the market.

1 “A Cosimulation Architecture for Power System, Communication, and Market in the Smart Grid” by Markus Mirz, Lukas Razik, Jan Dinkelbach, Halil Alper Tokel, Gholamreza Alirezaei, Rudolf Mathar, and Antonello Monti is licensed under CC BY 4.0

14 2.1 Fundamentals and Related Work

SGAM shall accelerate and standardize the development of unified data models, services, and applications in industry and research. In this context, the SINERGIEN data model and the co-simulation framework build upon SGAM as follows: • the unified data model formally defines the data exchange structure in alignment with the information layer concept of SGAM (see Sect. 2.4);

• the domain-speciﬁc simulators of our co-simulation environment include models of power grid and communication network components as well as market actors in the distribution, DER, and customer premise domains of the SGAM component layer;

• the communication layer is abstracted by a co-simulation interface and software extensions for the particular domain-speciﬁc simulators in order to enable data exchange between the components (see Sect. 2.4);

• the example use case presented in Sect. 2.2 with an optimal management of distributed battery storage systems is an example of a system function that would fall on the SGAM function layer. Fur- thermore, the business model motivating the provision of a proper system function, e. g., an incentive by a distribution system operator (DSO) is defined within the business layer. For our unified data model we chose CIM as well-established basis for power grid data that can be extended in a flexible manner. An extension of CIM was needed for communication infrastructure and energy market as for example shown in [Haq+11] and [Fre+09].

2.1.2 Common Information Model Some of the most important smart grid related standards (i. e. core standard) are from the IEC Technical Committee 57 (IEC TC 57). The so- called CIM is standardized in IEC 61970 (Energy Management Systems), IEC 61968 (Distribution Management), and IEC 62325 (Energy Market Communications) [IEC12b; IEC12a; IEC14]. Therefore, CIM belongs to the core standards included in the IEC/TR 62357 reference architecture [IEC; IEC16b].Originally CIM was developed as a database model for energy management systems (EMSs) and supervisory control and data acquisition (SCADA) systems but then changed into an object-oriented approach for electric distribution, transmission, and generation. Use cases of CIM are system integration using pre-deﬁned interfaces between the

15 Chapter 2 Multi-Domain Co-Simulation

IT of distribution management systems (DMSs) and automation parts, custom system integration using XML-based payloads for semantically sound coupling of systems, and serializing topology data using the Resource Description Framework (RDF) [Usl+12]. The IEC considers CIM and the IEC 61850 series as the pillars for a realization of the smart grid objectives of interoperability and device management [LE].

2.1.3 Simulation of Smart Grids

Example approaches for co-simulations on power grids and communication are presented in [Li+14; Hop+06; ZCN11; Lin+12] with focus on short- termed eﬀect and therefore not including the energy market. In MOCES [EFF15] a holistic approach is taken for modeling distributed energy systems but the result is a monolithic co-simulation and no co-simulation with a hybrid simulation for the physical part and an agent-based part for behavior-based simulations, e. g., coming from the market. With the SINERGIEN co-simulation environment the advantages of existing tools shall be harnessed which enhances the credibility of simulation results and obviate reinventing the wheel. The SINERGIEN co-simulation platform consists of several domain speciﬁc simulators with the possibility to use the “best tool” for each domain.

2.1.4 Classiﬁcation of Simulations

In [Sch+15] a classification scheme for energy-related co-simulations is introduced, with the four modeling categories continuous processes, discrete processes / events, roles, and statistical elements. The power grid in the SINERGIEN co-simulation environment is modeled based on Modelica.A short introduction to Modelica is provided in Sect. 4.1.1. Thermal systems [Mol+14] as well as power grids [MNM16] were modeled in Modelica. The Modelica models in the SINERGIEN co-simulation express continuous as well as discrete processes and events which makes the power grid simulation a hybrid simulation. The communication network is simulated with available discrete event simulation (DES) tools, such as ns-3. In a DES the simulation time does not proceed continuously but with the arising of certain events such as packet arrival, time expiry etc. [WGG10]. The energy market simulation was implemented also as DES but in Python which is flexible and suitable to test different optimization methods [Har+12]. Each market participant aims at optimizing the schedule for its assets, e. g., minimizing energy costs and maximizing its profit. Examples for statistical elements are, e. g., wind farm models of the power grid simulator.

16 2.2 Use Case

In view of the above, the SINERGIEN co-simulation environment is formalized as a coupled Discrete Event System Speciﬁcation (DES) as deﬁned in [ZPK00]. This formalization is shown in Sect. 2.4.

2.2 Use Case

The SINERGIEN environment can be used for an evaluation of diﬀerent scenarios with fast phenomena in the range from microseconds to seconds (i. e. with smaller simulation time steps) between highly dynamic power grid components, e. g., power electronics and communication network and slow phenomena in the range from minutes to hours (i. e. with larger simulation time steps) that include market entities, power grid, and communication network.

More on these two phenomena classes can be found in [Mir+18] with a focus on slow phenomena and a discussion on fast phenomena containing a description of adaptions needed for fast phenomena investigations. Based on this classification it can be concluded that the three simulators do not necessarily need to participate in each co-simulation. The example use case that was chosen in [Mir+18] for a validation of the SINERGIEN environment was an optimal management of distributed storage systems for peak-shaving to support the grid operation. The SINERGIEN environment including the communication network allows testing the effects of communication failures on the operation strategy and eventually on the electrical grid, which can provide valuable insights for decision making. Simulation results for this example are also provided in [Mir+18]. Before the co-simulation is initiated, it is necessary to define and store the topology under investigation along with the scenario-specific parameters. For example, various scenarios in which failures in the communication network are stochastically or deterministically set by the user in the data model can be examined. From a user perspective it would be advanta- geous if all components, their links, and parameters could be defined in one environment rather than splitting this information between different software solutions and formats. Then, the data model for the topology needs to include components that couple different domains. Under these requirements, following challenges were identified:

• deﬁnition of a common data model that includes components of all domains and their interconnections;

17 Chapter 2 Multi-Domain Co-Simulation

• interaction of simulators with diﬀerent simulation types, e. g., event- driven for the communication network and continuous processes for the power grid;

• choice of the co-simulation time step which is limited by the synchronization method connecting the simulators.

2.3 Challenges

A major issue in coupling of simulators with different modeling approaches is the selection and implementation of a synchronization mechanism which ensures proper progress of the simulation time and a timely data exchange between the simulators. This selection is of crucial significance for minimization of the error propagation in the co-simulation and the synchronization overhead in terms of simulation time. Since this is out of scope of this work, please refer to [Mir+18] for more details, whereas the definition and implementation of a new proper data model, involving all three mentioned domains, is crucial for the whole following work on large-scale co-simulation.

Holistic Topology Data Model

A common data model that covers the power grid, communication infrastructure and electrical market was not existent. Besides the benefit for the user of a co-simulation environment with a single data model, for the specification of a holistic co-simulation topology, also the data exchange between simulators is simplified. A system description that encompasses all components of smart grids as shown in Fig. 2.1 (1) can be either used directly by a single multi-domain smart grid simulator or divided into subsystems for a co-simulation as in Fig. 2.1 (2). For many components, this division is obvious since their parameters are only needed by one domain-specific simulator but some components (called inter-domain components) constitute natural coupling points between the three domains. For instance, a battery storage device connected to the grid can act as a market participant that offers its capability to charge or discharge. In order to enable its participation in the energy market, the battery storage needs an interface which is a communication modem in this case. The modem can be seen as a part of the battery storage. For a co-simulation, the information on inter-domain components must be split into several parts as each simulator has to simulate a dedicated part of these components.

18 2.4 Concept of the Co-Simulation Environment

2.4 Concept of the Co-Simulation Environment

2.4.1 Holistic Topology Data Model

As already mentioned, a holistic data model for a whole three domains co-simulation topology can be based on CIM with an extension by further classes. These classes, introduced for completing CIM in its representation of smart grids, are linked to already existing CIM classes using the Uniﬁed

Wholesale and Retail Market

Market Participants

(1)

Wholesale and Retail Market

Market Participants

Communication Power Grid Market Network (2)

Figure 2.1: Exemplary topology including components of (1) all domains and (2) domain-speciﬁc topologies

19 Chapter 2 Multi-Domain Co-Simulation

Modeling Language (UML). The proposed format can be structured in four packages: • Original CIM (IEC 61970 / 61968 / 62325),

• Communication,

• Market, and

• EnergyGrid. Whenever suitable, original CIM classes are considered. However, some components do not have an associated class in the standard yet and therefore are added in one of the other three packages. This approach leads to a ﬂexible update to a new CIM version without losing the added classes with their links. The most important feature of the SINERGIEN data model is the interconnection of domains. Examples of inter-domain components, namely BatteryStorage, SolarGeneratingUnit, and MarketCogeneration, are shown in Fig. 2.2, an excerpt from the SINERGIEN data model. According to the UML diagram, the energy market components are associated to the power grid components, whereas power grid components have an

Market

Equipment:: Equipment:: Equipment:: MarketBatteryStorage MarketSolarGeneratingUnit MarketCogenerationUnit

Power Grid RegulatingCondEq GeneratingUnit PowerSystemResource

EnergyStorage:: Production:: Production:: BatteryStorage SolarGeneratingUnit CogenerationPlant

Communication

Modems::ComMod

Figure 2.2: Inter-domain connections between classes of power grid, communication network and market

20 2.4 Concept of the Co-Simulation Environment aggregation relationship to communication devices. This means that parameters specific to the market, communication network, and power grid which relate to the same device are linked with each other. Therefore, all information on one device is easily accessible but at the same time there is a separation according to the domains. The connections between classes of different domains are defined in a logical and not a topological manner. Instead, topological connections exist to interconnect power grid components, for instance. In the mentioned battery storage device example, the data model is as follows: the device is a part of the grid and has electrical parameters. Furthermore, the battery storage might participate in the market, e. g., as part of a virtual power plant (VPP). Market-specific information can be stored in MarketBatteryStorage class objects which is associated with the BatteryStorage. The communication modem ComMod which could be used to communicate with the VPP is aggregated to the BatteryStorage class. The three additional packages EnergyGrid, Communication, and Market are needed for the following: • some newer components occurring in power grids are missing in original CIM. For instance, it was necessary to create a new model for electrical energy storages like stationary batteries. A battery storage is a conducting equipment that is able to regulate its energy through- put in both directions. Therefore, the class BatteryStorage added in the EnergyGrid package is a specialization of a CIM Regulating- ConductingEquipment since it can influence the flow of power at a specific point in the grid.

• the key component of the Market package for the scenarios that we would like to investigate is a VPP since the aggregation of small DER units enables their participation at electricity markets.

• the Communication package includes all additionally deﬁned classes that are related to the communication network model, such as classes for communication links and technologies, modems, network nodes along with their parameters and their relations with the classes in CIM, power grid package, and market package. Figure 2.3 shows an excerpt from the communication data model with an aggregation to a WindGeneratingUnit. By means of the associated classes for modems, communication requirements and channels, the model enables a description of network parameters and topology. More on the packages can be found in [Mir+18].

21 Chapter 2 Multi-Domain Co-Simulation

2.4.2 Model Data Processing and Simulation Setup

The overall information ﬂow for the simulation setup is depicted in Fig. 2.4. After the holistic topology is edited in a graphical Topology Builder, including all objects of the three domains, it is forwarded to the co- simulation interface. In order to execute a simulation, the Modelica solver requires a Modelica model, whereas the communication network topology can be given to the communication network simulator in CIM format, which includes the components of the network, their connections, and parameters. The co-simulation interface incorporates a component called CIMverter, based on CIM++ [Raz+18a] presented in Chap. 3. The CIMverter [Raz+18b] reads in the CIM document and outputs a Modelica system model (Chap. 4) for the power grid simulator. In contrast, the Python-based market simulation relies on a C++/Python interface, which could be realized using one of the common libraries for Python to wrap C++ data types and functions, to retrieve the market relevant information from the C++ objects and store them in Python objects. A detailed explanation of the translation from CIM to Modelica is given in Chap. 4.

GeneratingUnit WindGeneratingUnit 1 1 0..* 0..1

ComMod CommunicationRequirement

WirelessMod WiredMod

WiredInterface LTEModem FiberModem FiberInt 1 1..* 1 1..* WiredChannel FiberChannel

Figure 2.3: Communication network class association example

22 2.4 Concept of the Co-Simulation Environment

2.4.3 Synchronization

The synchronization during simulation is performed at fixed time steps. For slow phenomena scenarios this is managed by mosaik, a well-established co-simulation framework [SST11]. It allows coupling the three simulators in a simple manner as explained in Sect. 2.4.4 in case of longer synchronization time steps. VILLASnode, a software project for coupling real-time simulations in LANs [Vog+17; Ste+17], is a suitable alternative for mosaik in the case of synchronizations with very short synchronization time steps. In Modelica, the synchronization data exchange is achieved by integrating Modelica blocks of the Modelica_DeviceDrivers library, which was originally developed for interfacing devices to Modelica environments [Thi19]. The library conveniently allows the definition of a fixed interval for data exchange that can be different from the simulation time step. More on this choice and the integration can be found in [Mir+18]. Figure 2.5 depicts the flow of time for the co-simulation and each simulator. The power grid and market simulators compute in parallel, whereas the com-

Block diagram Topology Builder of topology

Extended CIM XML representation of topology

Co-Simulation Interface mosaik, CIM++, CIMverter, etc. C++ objects from CIM++ & simulator-speciﬁc system models

Communication Modelica Models Python Objects Network Topology

Power Grid Communication Simulation Market Simulation Simulation Modelica (Dymola, Python ns-3, etc. OpenModelica)

Figure 2.4: Overall SINERGIEN architecture for simulation setup

23 Chapter 2 Multi-Domain Co-Simulation munication network is waiting for their inputs. The interactions between the simulators in each co-simulation step can be formalized by

up(n + 1) = Fc(Fm(um(n))), (2.1)

um(n + 1) = Fc(Fp(up(n))), (2.2) where uc, um and up are the corresponding input values of the simulators for the power grid, energy market and communication network for each time step. Therefore, it is required to set initial values, up(0), um(0), uc(0), at the beginning of the co-simulation. n denominates the current co-simulation time step. Fc (communication), Fm (market) and Fp (power grid) are the functions describing the calculation within a step.

2.4.4 Co-Simulation Runtime Interaction Figure 2.6 shows the coupling of the simulators for their co-simulation runtime interaction with following entities: mosaik As already mentioned, mosaik is used for the coordination during the synchronization steps of several minutes (in simulation time) regarding all simulators [Sch19]. Market Simulator Implemented in Python, it can make use of mosaik’s so-called high-level API as illustrated in Fig. 2.6. Communication Network Simulator Based on available DES tools, their network simulation modules are extended with inter-process communication functionalities for message exchange with mosaik.

0 1 1 2

Power Power Power Grid Grid Grid

Communication Communication Network Network

Market 0 1 Market 1 2 Market Event 0 Event 2

0 1 Event 1 1 2 Event 3 Individual Simulator Steps Co-Simulation 0 1 2 Steps

Figure 2.5: Synchronization scheme of simulators at co-simulation time steps

24 2.4 Concept of the Co-Simulation Environment

Power System Simulator The integration of so-called TCPIP_Send/ Recv_IO blocks from Modelica_DeviceDrivers into the Modelica models, allows the exchange of simulation data via sockets but in the form of Modelica variables as bitvectors instead of messages in JSON, an open-source and human-readable data format [ecm19]. Therefore, the MODD Server is implemented.

MODD Server It receives commands from the socket connected with mosaik. Based on these commands it starts, for example, the power

Python Environment

Market Simulation

Slow phenomena communication TCP Sockets

TCP Sockets TCP Sockets TCP Sockets

mosaik- MODD Server core Communication TCP Sockets Simulation

DES TCP Sockets Environment Modelica InﬁniBand DeviceDrivers

InﬁniBand Power Grid Simulation VILLAS- node Modelica Shared Memory Shared Memory DeviceDrivers Modelica Environment Fast phenomena communication

Figure 2.6: Scheme of runtime interaction between co-simulation components

25 Chapter 2 Multi-Domain Co-Simulation

grid simulator or receives the bitstream from Modelica_DeviceDrivers and encapsulates it into JSON messages before transferring them to mosaik. Besides the synchronization steps controlled by mosaik, there will be also more ﬁne-grained synchronization steps of fractions of seconds between the power grid and communication network simulator. That is why a VILLASnode gateway is included.

VILLASnode Instead of the Transmission Control Protocol (TCP) as in case of mosaik, VILLASnode can make use of InﬁniBand (IB) interconnects for data exchange between real-time simulators on diﬀerent machines and shared-memory regions on the same machine. The use of shared-memory regions and IB interconnects leads to lower latencies and consequently to shorter synchronization time steps as shown in Chap. 8.

For more on the formalization of the SINERGIEN co-simulation and the limitations of the environment please refer to [Mir+18].

2.5 Validation by Use Case

The proper functioning of the SINERGIEN co-simulation environment has been validated with the aid of diﬀerent use case scenarios. In the use case presented in [Mir+18] there is the assumption that a VPP operator tries to reduce the VPP’s peak power. This behavior could be desired by the responsible DSO and come with ﬁnancial incentives. Therefore, a peak- shaving algorithm is utilized for an optimal management of distributed battery storage systems. First of all, simulation results which are obtained without the SIN- ERGIEN environment were compared with results obtained with the SIN- ERGIEN environment for demonstrating that the results do not change under the assumption of an ideal communication network when simulating the same scenario. Furthermore another scenario was presented where the communication network was supposed to impair the control loop between the power grid and the market due to communication device failures. More on the details of the co-simulated scenarios again can be found in [Mir+18] as the simulations themselves are not in focus of this work. Anyway, with the simulation results of both scenarios a proper functioning of the co-simulation environment has been shown.

26 2.6 Conclusion

2.6 Conclusion

The here presented architecture of the implemented multi-domain co- simulation environment shows the applicability of the CIM-based holistic data model for smart grid simulations which include the three domains: power grid, communication network, and market. The data model facili- tates the use of the software environment since the domain-specific smart grid component parameters and their interconnections can be modified and stored in a self-contained topology description. Due to the SINERGIEN co-simulation approach the user can take advantage of established domain- specific simulators for each domain. For this purpose, also new software tools have been developed. The ModPowerSystem library can be used for scientific research on various models since Modelica as modeling language simplifies the development and improvement of component models. Because of the increasing use of CIM- based documents for grid topology representation, the choice of Modelica lead to the development of a CIM to Modelica mapping that is presented in Chap. 4. Besides the initiation of the CIM related topics (Chap. 3 and Chap. 4), the SINERGIEN co-simulation architecture illustrates how the work on HPC Python (Chap. 7) and the integration of InfiniBand in VILLAS Chap. 8 can be used in power system co-simulation. The work in Chap. 5 and Chap. 6, however, contributes to a higher performance of the simulation itself which is accomplished by the simulators of the co-simulation environment. In the following chapter, the automated generation of a (de-)serializer for reading and writing CIM-based documents, implemented in the mentioned CIM++ software library, is presented.

Automated De-/Serializer Generation

Due to growing automation in smart grids with the aid of an increasing digitalization and a rising number of decentralized energy systems, the actors in this area are increasingly dependent on ICT systems that must be compatible with each other, which in particular concerns the data exchange between these systems. Therefore, different countries, organiza- tions, and vendors started to develop smart grids related standards with different focus on technical and economical aspects. Eventually, only few national standards have been integrated into standards of the International Electrotechnical Commission (IEC) or the International Organization for Standardization (ISO) [Usl+12]. In recent years, the CIM standards (IEC 61970/61968/62325, see Sect. 2.1.2) have been subject to numerous research activities often related to use cases for CIM [MDC09; DK12; Wei+07]. Some of them, like in the research project SINERGIEN, also introduce extensions by classes not included in original CIM as, for instance, in [MMS13] where a methodology for modeling telemetry in power systems using IEC 61970/68 in the case of a US independent system operator is presented. There are also harmonization approaches because of data exchange between energy related software systems based on CIM standards and ICT for substation automation based on IEC 61850 [LK17; Bec10; SRS10]. Since CIM is object-oriented, it specifies classes of objects containing information about energy system aspects as well as relations between these classes (referred to as the ontology)[GDD+06]. Currently more

29 Chapter 3 Automated De-/Serializer Generation and more commercial software tools in the energy sector provide import and export of CIM documents. Moreover, there are already about 200 corporate members organized in the CIM User Group (CIMug) providing CIM models for common visual Unified Modeling Language (UML) editors [CIM]. This high acceptance among companies and institutions has pushed the adoption of CIM also in the simulation environment as presented in Chap. 2 with respect to the SINERGIEN co-simulation environment, where the data format of the multi-domain component-based co-simulation model (referred to as the topology) is based on CIM. As the topology, including power grid and communication network components as well as energy market actors, evolves continuously, a high compatibility, updatability, and extensibility of the chosen data model are key requirements. The object-oriented design with concepts such as inheritance, associations, aggregations, etc. led to a CIM data encapsulation format referred to as RDF/XML [IEC06] coming from the area of semantic web [AH11] and not as common in other domains. This and the huge specification of CIM with hundreds of classes and relations between them, making CIM very extensible and universal applicable in comparison to other more specific and static data models, have a deterrent effect to new users. However, keeping CIM based software up-to-date continuously can be too effortful, especially in the scientific and academic area. These could be the reason why there are hardly any software libraries especially for handling CIM documents. Therefore, in this chapter an automated (de-)serializer generation from UML based CIM ontologies is presented. The approach was introduced in [Raz+18a] and implemented in a chain of tools for generating an open- source library libcimpp within the CIM++ software project [FEI19a]. libcimpp can be used for reading in CIM RDF/XML documents directly into CIM C++ objects (called deserialization) and is currently also extended for serialization (i. e. writing of CIM C++ objects from the memory to RDF/XML documents). Due to a model-driven architecture (MDA), libcimpp can be adapted to new CIM versions or user-specified CIM based ontologies in an automated way. For this purpose, the approach makes use of a common visual UML editor and our CIM++ toolchain generating a complete compilable CIM C++ codebase of given CIM UML models (i. e. CIM profiles) which are kept up-to-date (e. g. by the CIMug). It is also shown how this CIM C++ codebase can be used for holding the deserialized CIM objects as well as for an automated generation of C++ code for exactly this deserialization. Hence, if the CIM C++ codebase changes (because of changes in the CIM UML), there is no need to adapt code for libcimpp by hand.

30 3.1 CIM Formalisms and Formats

The direct deserialization into C++ objects makes the library very easy to apply because its user does need neither any CIM RDF/XML knowledge nor have to handle intermediate representations of the CIM RDF/XML document like a Document Object Model (DOM) in combination with the Resource Description Framework (RDF) syntax. For instance, in case of a power grid topology stored in CIM documents, a power grid simulator can directly access the CIM objects, deserialized by libcimpp, in form of common C++ objects. The chapter gives a short introduction to data formats as well as other components used in CIM++ followed by an overview of the overall concept. Then it explicitly describes how Common Information Model (CIM) is au- tomatedly mapped to compilable C++ code which is used by the CIM++ Deserializer (i. e. libcimpp) during the so-called unmarshalling step explained subsequently together with its automated generation. Following this, the ﬁnal libcimpp is introduced. Finally, the chapter is concluded by a roundup and an outlook of future work. The work in this chapter has been partially presented in [Raz+18a]1.

3.1 CIM Formalisms and Formats

An introduction to CIM is provided by Sect. 2.1.2. CIM makes use of several formalisms and formats which are explained in the following.

UML UML is a well-established formalism for graphical object-oriented modeling [RJB04]. In CIM only UML class diagrams with attributes and inheritance as well as associations, aggregations, and compositions with multiplicities are used. The CIM UML contains no class methods as CIM deﬁnes just the semantics of its object classes and their relations without any functionality of the objects just to specify which kind of information a CIM object contains. CIM UML diagrams can, as other UML diagrams, be edited by visual UML editors and stored in a proprietary or open format like XML Metadata Interchange (XMI) [KH02]. Conveniently, the CIMug provides such CIM model drafts [CIM]. While UML resp. XMI is used for the deﬁnition

1 Reprinted by permission from Springer Nature Customer Service Centre GmbH: Springer Computer Science – Research and Development (“Automated deserializer generation from CIM ontologies: CIM++—an easy-to-use and automated adaptable open-source library for object deserialization in C++ from documents based on user-speciﬁed UML models following the Common Information Model (CIM) standards for the energy sector“, Lukas Razik, Markus Mirz, Daniel Knibbe, Stefan Lankes, Antonello Monti), © (2017)

31 Chapter 3 Automated De-/Serializer Generation of all classes with their attributes and relations among them, the actual objects (i. e. instances of these classes) are stored in form of RDF/XML documents.

XML and RDF

The Extensible Markup Language (XML) is a widely used text-based formalism for human- and machine-readable documents [Bra+97]. In general, XML documents have a tree structure which is why XML itself is not well suited for representing arbitrary graphs. Therefore it is combined with the RDF [Pan09]. RDF provides triples of the form “