ALMA MATER STUDIORUM – UNIVERSITY OF BOLOGNA Ph.D. THESIS FACULTY OF ENGINEERING PH.D. PROGRAM IN ELECTRONICS ENGINEERING, TELECOMMUNICATIONS AND INFORMATION TECHNOLOGY Hardware/Software Design of Dynamic Real-Time Schedulers for Embedded Multiprocessor Systems AUTHOR ADVISOR Primiano Tucci Prof. Antonio Corradi CO-ADVISOR Prof. Eugenio Faldella PH.D. COORDINATOR Prof. Alessandro Vanelli Coralli Ciclo: XXV – A.A.: 2011/12 – SSD: ING-INF/05 – SETTORE CONCORSUALE: 09/H1 [This page intentionally left blank] 2 Abstract The new generation of multicore processors opens new perspectives for the design of embedded systems. Multiprocessing, however, poses new challenges to the scheduling of real-time applications, in which the ever- increasing computational demands are constantly flanked by the need of meeting critical time constraints. Many research works have contributed to this field introducing new advanced scheduling algorithms. However, despite many of these works have solidly demonstrated their effectiveness, the actual support for multiprocessor real-time scheduling offered by current operating systems is still very limited. This dissertation deals with implementative aspects of real-time schedulers in modern embedded multiprocessor systems. The first contribution is represented by an open-source scheduling framework, which is capable of realizing complex multiprocessor scheduling policies, such as G-EDF, on conventional operating systems exploiting only their native scheduler from user-space. A set of experimental evaluations compare the proposed solution to other research projects that pursue the same goals by means of kernel modifications, highlighting comparable scheduling performances. The principles that underpin the operation of the framework, originally designed for symmetric multiprocessors, have been further extended first to asymmetric ones, which are subjected to major restrictions such as the lack of support for task migrations, and later to re-programmable hardware architectures (FPGAs). In the latter case, this work introduces a scheduling accelerator, which offloads most of the scheduling operations to the hardware and exhibits extremely low scheduling jitter. The realization of a portable scheduling framework presented many interesting software challenges. One of these has been represented by timekeeping. In this regard, a further contribution is represented by a novel data structure, called addressable binary heap (ABH). Such ABH, which is conceptually a pointer-based implementation of a binary heap, shows very interesting average and worst-case performances when addressing the problem of tick-less timekeeping of high-resolution timers. 3 [This page intentionally left blank] 4 “Life’s a trampoline. Sometimes it takes someone trying to bring you down, in order to jump higher” Santa Clara, CA 4 Oct 2012 5 Preface Many things have changed in my life in these years as a PhD student. In the last year, in particular, I had the unique opportunity of working in two of the coolest tech companies of our times, respectively Google Inc. and NVIDIA Corp., that, for a curious turn of events, I joined with near- perfect timing. In the former, I had the pleasure to join the Chrome for Android team while the most advanced mobile browser of the time was firstly launched and soon thereafter became the official browser of the Android OS. In the latter, I had the amazing opportunity to join the software engineering team in the months in which the NVIDIA Tesla K20 debuted as the GPGPU powering the most powerful supercomputer of the world 1, the Cray XK7 Titan. Things will unavoidably change in the future years and perhaps one day many of us will smile rereading these lines and thinking back to them with the same nostalgic mood that we have today seeing screenshots of NCSA Mosaic or reproductions of the Cray-1 in museums. However, one thing will remain the same, whatever changes the future will bring: the thought that I was there and was actively contributing to them. Many of these changes could not have happened without the influence of many people who crossed my path in these years. A grateful acknowledgement, therefore, goes to them, for the spiritual and material support provided. First of all I would like to thank all the many friends that I had the pleasure and the luck to meet during my early university years. I will never forget all the pleasant moments we shared together, the uncountable dinners and the never-ending nights of our carefree student years spent in the old streets of Bologna. A further special thank goes to my friends and colleagues Andrea, Giuseppe and Mario, with whom I had the pleasure to share this PhD adventure (and the unforgettable flavors of the campus’ cafeteria). I would like to express my earnest gratitude to my PhD advisor Prof. Eugenio Faldella, for his remarkable wisdom, honesty, enthusiasm and 1 http://www.top500.org/lists/2012/11/ 6 friendliness (and, lastly, passion for brain teaser games), which I had the pleasure to appreciate during the numerous days (and late-evenings) spent working together. Some of my acknowledgements go also to companies. In first place, IMA S.p.A. and SACMI Imola, who believed and supported my work and with which I had the pleasure to collaborate, in my early years, on very challenging embedded industrial automation projects. In particular, my thanks goes for the rewarding thought that parts of this thesis will continue to live in many of the tea bags produced worldwide. Another thanks goes to the B&R automation and Altera corporations, for the interests that they have demonstrated in the academy through the time and the resources dedicated to university contests, which I attended with some of the projects herein presented (see next page). A remarkable acknowledgement finally goes to the amazing people I had the opportunity to work with during my last internships. To them goes my greatest and most sincere admiration. Besides their unquestionable and unequaled talent, their wisdom and their technical skills, I have been gladly impressed by the great character and charm of many of them that with extreme discretion, great passion and hard commitment contribute to the technological evolution of our world. A particular thanks, in this regard, goes to my mentors Hans and Lucien, who helped me getting started into these two amazing companies and had the patience to endure my bazillion questions and review my code. Finally yet importantly, I would like to thank my family that have supported and motivated me during all these years. To them goes my constant and profound gratitude, especially in these months of oceanic distance. 7 Altera InnovateItaly 2011 “Home automation over mains with Altera NIOS-II” First place. B& R European Industrial Ethernet Award 2010/11 “Powerlink over PowerLine: the next generation of home automation runs in real-time over mains” Winner in the category 'relevance for the industry'. Altera InnovateItaly 2010 “Hard real-time meets embedded multicore SoPCs” First place, ex-aequo. 8 Table of Contents Abstract ..................................................................................................... 3 Preface ...................................................................................................... 6 Table of Contents ...................................................................................... 9 1. Introduction......................................................................................... 11 1.1. Scenario and motivations ............................................................. 11 1.2. Contributions ................................................................................ 12 1.3. Organization ................................................................................. 14 2. Background and related work ............................................................. 17 2.1. Embedded real-time systems ....................................................... 17 2.2. Real-time task model ................................................................... 19 2.3. The uniprocessor real-time scheduling problem .......................... 21 2.4. Taxonomy .................................................................................... 23 2.5. Uniprocessor real-time scheduling............................................... 25 2.6. Multiprocessor systems ................................................................ 27 2.7. Multiprocessor real-time scheduling ............................................ 32 2.8. Real-Time operating systems ....................................................... 38 2.9. IEEE POSIX standards for real-time applications ....................... 44 2.10. Linux as a real-time operating system ....................................... 52 3. X-RT: A portable framework for real-time scheduling ...................... 61 3.1. Introduction .................................................................................. 61 3.2. Motivations .................................................................................. 61 3.1. Related work ................................................................................ 62 3.2. Software architecture for SMP ..................................................... 65 3.3. Implementation of the G-EDF scheduling policy ........................ 83 3.4. SMP experimental evaluations. ................................................... 86 3.5. Software architecture for AMP .................................................. 103 3.6. AMP experimental