Parallel SystemC Simulation for Electronic System Level Design Von der Fakultät für Elektrotechnik und Informationstechnik der Rheinisch–Westfälischen Technischen Hochschule Aachen zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften genehmigte Dissertation vorgelegt von Diplom–Ingenieur Jan Henrik Weinstock aus Göttingen Berichter: Universitätsprofessor Dr. rer. nat. Rainer Leupers Universitätsprofessor Dr.-Ing. Diana Göhringer Tag der mündlichen Prüfung: 19.06.2018 Diese Dissertation ist auf den Internetseiten der Hochschulbibliothek online verfügbar. Abstract Over the past decade, Virtual Platforms (VPs) have established themselves as essential tools for embedded system design. Their application fields range from rapid proto- typing over design space exploration to early software development. This makes VPs a core enabler for concurrent HW/SW design – an indispensable design approach for meeting today’s aggressive marketing schedules. VPs are essentially a simulation of a complete microprocessor system, detailed enough to run unmodified target binary code. During simulation, VPs provide non-intrusive debugging access as well as re- porting on non-functional system parameters, such as execution timing and estimated power and energy consumption. To accelerate the construction of a VP for new systems, developers typically rely on pre-existing simulation environments. SystemC is a popular example for this and has become the de-facto reference for VP design since it became an official IEEE stan- dard in 2005. Since then, however, SystemC has failed to keep pace with its user’s demands for high simulation speed, especially when embedded multi-core systems are concerned. Because SystemC only utilizes a single processor of the host computer, the underlying sequential discrete event simulation algorithm becomes a performance bottleneck when simulating multiple virtual processors. It is the main goal of this thesis to overcome this bottleneck by utilizing multi-core computers as simulation hosts, given they are abundantly available today. To that ex- tent, it presents new tools and modelling methodologies that facilitate parallelization of new and existing VPs, even if they include legacy source code and models that were written without parallelism or thread-safety in mind. After proposing a new parallel simulation engine for SystemC, this thesis continues by investigating modelling ap- proaches that address commonly encountered issues when migrating from sequential to parallel simulation environments, such as the functional correct representation of optimistic exclusive memory access models. The efficacy of the proposed techniques is evaluated using realistic VPs as the driving case studies. In the past, these platforms have been used productively, either in design space exploration or as debug targets for early SW development. Their performance gains due to parallel simulation reach 4 – 8x over the current state-of- the-art implementation of SystemC on modern multi-core host computers. Acknowledgements While writing these words I am looking back at the six years (even more if you count my time as a student and research assistant) I spent at the Institute for Communication Technologies and Embedded Systems at the RWTH Aachen University. It has truly been a privilege to be able to work alongside so many excellent personalities that have supported and shaped my academic career and I would like to take the opportunity to thank them here. First, I would like to thank Professor Rainer Leupers, who encouraged me to pursue a doctoral degree and allowed me to join his research group. His vast expertise and constructive feedback allowed me to refocus my research whenever I lost track and helped me to concentrate on solving the practical problems. I am grateful to Christoph Schumacher, who accompanied me during my first voyages into SystemC and parallel simulators during my early days at the Institute. His determined and thorough approach to problem solving quickly became a shining example and his research formed a solid foundation to build upon. Beyond those already mentioned, I had the pleasure of working with many other Institute members that deserve a mention here. I would like to express my gratitude to Luis Gabriel Murillo, Róbert Lajos Bücs and Stefan Schürmans for our inspirational conversations as well as challenging questions that helped me improving and opti- mizing this work. I would also like to thank my research assistants Christian Jöhri, Alexander Wihl and Florian Walbroel for helping me to carry the workload. A special thank you goes to my colleagues Gereon Onnebrink and Diego Pala for proof-reading and helping me to give this thesis its finishing touch. Finally, I would like to express my gratitude to my friends and family – those who are still around and those who passed away too soon. It is because of your love and support that this work became possible, and I cannot thank you enough for it. Jan Henrik Weinstock, June 2018 Contents 1 Introduction 1 1.1 EmbeddedSystems............................... 2 1.2 ElectronicSystemLevelDesign . ... 4 1.3 SystemLevelDescriptionLanguages . .... 5 1.3.1 SystemC ................................. 7 1.3.2 SpecC................................... 7 1.3.3 SystemVerilog .............................. 8 1.3.4 Summary................................. 8 1.4 AnatomyofaVirtualPlatform . 9 1.4.1 SupportingToolset ........................... 9 1.4.2 PlatformSimulator ........................... 11 1.5 ThesisContributionsandOutline. .... 12 1.6 Synopsis ..................................... 13 2 Background 15 2.1 DiscreteEventSimulation . .. 15 2.2 DeterministicSimulation. ... 18 2.3 RaceConditions................................. 19 2.4 SynchronisationProblem. .. 21 2.5 Synopsis ..................................... 24 3 Related Work 25 3.1 TraditionalParallelSimulation . ..... 25 3.2 ParallelSystemC ................................ 26 i ii CONTENTS 3.2.1 SynchronousSimulation . 26 3.2.2 AsynchronousSimulation . 27 3.2.3 DistributedSimulation . 28 3.2.4 AcceleratorSupportedSimulation . .. 29 3.2.5 SummaryofParallelSystemCApproaches . 30 3.3 FastInstructionSetSimulation . .... 32 3.3.1 CompiledSimulation. 33 3.3.2 DynamicBinaryTranslation. 34 3.4 Synopsis ..................................... 34 4 Target Platforms 35 4.1 TheEURETILEPlatform ............................ 35 4.1.1 VirtualEURETILEPlatform . 36 4.1.2 EURETILESoftwareandToolchain. 37 4.2 TheGEMSCLAIMPlatform . .. .. .. .. .. .. .. 38 4.2.1 GEMSCLAIMVirtualPlatform . 39 4.2.2 GEMSCLAIMSoftwareEnvironment . 41 4.3 TheOpenRISCPlatform ............................ 41 4.3.1 OpenRISCVirtualPlatform . 42 4.3.2 OpenRISCSoftwareEnvironment . 43 4.4 Synopsis ..................................... 44 5 Time-Decoupled Parallel SystemC 45 5.1 AsynchronousSystemCSimulation. ... 45 5.1.1 SimulationLoopandContext . 47 5.1.2 SimulatorPartitioning . 48 5.2 Cross-threadCommunication . .. 50 5.2.1 RemoteEvents.............................. 51 5.2.2 RemoteEventQueues . .. .. .. .. .. .. .. 53 5.2.3 BlockingTransportInterface. .. 55 5.2.4 AugmentedTLMTargetSocket . 57 5.3 ExperimentalResults . .. .. .. .. .. .. .. 59 CONTENTS iii 5.3.1 ExperimentSetup............................ 60 5.3.2 SequentialPerformance . 62 5.3.3 ParallelPerformance . 63 5.3.4 LookaheadAnalysis . .. .. .. .. .. .. .. 65 5.4 LimitationsandOutlook . 66 5.5 Synopsis ..................................... 68 6 Flexible Time-Decoupling 69 6.1 SimulatorOperationModes . 69 6.1.1 DeterministicSimulationMode . 71 6.1.2 FastSimulationMode . 72 6.2 FlexibleInter-threadCommunication . ..... 74 6.2.1 Zero-delayRemoteNotifications . 74 6.2.2 RemoteSignals ............................. 76 6.2.3 RemoteDirectMemoryAccess . 77 6.3 TemporalDecoupling. .. .. .. .. .. .. .. .. 78 6.3.1 TimingError............................... 78 6.3.2 MitigationStrategy .. .. .. .. .. .. .. .. 79 6.4 ExperimentalResults . .. .. .. .. .. .. .. 79 6.4.1 SyntheticExperiments . 80 6.4.2 GEMSCLAIMExperimentSetup . 82 6.4.3 GEMSCLAIMExperimentation . 83 6.5 LimitationsandOutlook . 85 6.6 Synopsis ..................................... 86 7 Exclusive Memory Access Models 87 7.1 ModellingConsiderations . .. 88 7.1.1 Load-LinkedandStore-Conditional . .. 88 7.1.2 TheABAProblem ........................... 90 7.2 ModellingApproach .............................. 91 7.2.1 DMICacheModel ........................... 92 7.2.2 Memory-basedModel . 93 iv CONTENTS 7.2.3 Transaction-basedModel . 95 7.2.4 MixedOperation ............................ 97 7.3 ExperimentalResults . .. .. .. .. .. .. .. 99 7.3.1 SimulationOverhead. 100 7.3.2 ParallelPerformance . 102 7.4 LimitationsandOutlook . 105 7.5 Synopsis ..................................... 106 8 Processor Sleep Models 107 8.1 ProcessorSleepStates . .. .. .. .. .. .. .. 108 8.2 TheOpenRISCInstructionSetSimulator . 109 8.2.1 CacheCompiledSimulation. 111 8.2.2 SystemCWrapper. .. .. .. .. .. .. .. .. 112 8.3 ProcessorSleepModels. 113 8.3.1 ISSSleepModel............................. 113 8.3.2 DESSleepModel ............................ 115 8.4 ExperimentalResults . .. .. .. .. .. .. .. 117 8.4.1 ProcessorActivityTracing . 118 8.4.2 SleepModelComparison . 119 8.4.3 ParallelPerformanceAnalysis . 121 8.5 LimitationsandOutlook . 123 8.6 Synopsis ..................................... 124 9 ParallelSystemCusingTime-DecoupledSegments 125 9.1 TheSystemC-LinkConcept . 126 9.2 SimulationController. 127 9.2.1 Time-DecoupledCo-Simulation. 128 9.2.2 SegmentScheduling . 129 9.2.3 VirtualSequentialEnvironment