A Comparison of Artificial Intelligence Algorithms for Dynamic Power Allocation in Flexible High Throughput Satellites by Juan Jose Garau Luis Submitted to the Department of Aeronautics and Astronautics in partial fulfillment of the requirements for the degree of Master of Science in Aeronautics and Astronautics at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY May 2020 ○c Massachusetts Institute of Technology 2020. All rights reserved. Author................................................................ Department of Aeronautics and Astronautics May 13, 2020 Certified by. Prof. Edward F. Crawley Professor of Aeronautics and Astronautics Thesis Supervisor Accepted by . Sertac Karaman Associate Professor of Aeronautics and Astronautics Chair, Graduate Program Committee 2 A Comparison of Artificial Intelligence Algorithms for Dynamic Power Allocation in Flexible High Throughput Satellites by Juan Jose Garau Luis Submitted to the Department of Aeronautics and Astronautics on May 13, 2020, in partial fulfillment of the requirements for the degree of Master of Science in Aeronautics and Astronautics Abstract The Dynamic Resource Management (DRM) problem in the context of multibeam satellite communications is becoming more relevant than ever. The future landscape of the industry will be defined by a substantial increase in demand alongside the introduction of digital and highly flexible payloads able to operate and reconfigure hundreds or even thousands of beams in real time. This increase in complexity and dimensionality puts the spotlight on new resource allocation strategies that use autonomous algorithms at the core of their decision-making systems. These algorithms must be able to find optimal resource allocations in real or near-real time. Traditional optimization approaches no longer meet all these DRM requirements and the research community is studying the application of Artificial Intelligence (AI) algorithms to the problem as a potential alternative that satisfies the operational constraints. Although multiple AI approaches have been proposed in the recent years, most of the analyses have been conducted under assumptions that do not entirely reflect the new operation scenarios’ requirements, such as near-real time performance or high- dimensionality. Furthermore, little work has been done in thoroughly comparing the performance of different algorithms and characterizing them. This Thesis considers the Dynamic Power Allocation problem, a DRM subproblem, as a use case and compares nine different AI algorithms under the same near-real time operational assumptions, using the same satellite and link budget models, and four different demand datasets. The study focuses on Genetic Algorithms (GA), Simulated Annealing (SA), Particle Swarm Optimization (PSO), Deep Reinforcement Learning (DRL), and hybrid approaches, including a novel DRL-GA hybrid. The comparison considers the following characteristics: time convergence, con- tinuous operability, scalability, and robustness. After evaluating the algorithms’ performance on the different test scenarios, three algorithms are identified as potential candidates to be used during real satellite operations. The novel DRL-GA implemen- tation shows the best overall performance, being also the most robust. When the 3 update frequency is in the order of seconds, DRL is identified as the best algorithm, since it is the fastest. Finally, when the online data substantially diverges from the training dataset of the DRL algorithm, both DRL and DRL-GA hybrid might not perform adequately and an individual GA might be the best option instead. Thesis Supervisor: Prof. Edward F. Crawley Title: Professor of Aeronautics and Astronautics 4 Acknowledgments This Thesis has not been written under normal circumstances. Right now the world is suffering the social, economic, and structural effects of the COVID-19 global pandemic. My first thoughts of gratitude go to all healthcare workers on the front line andpeople carrying out essential tasks that keep our society moving. I also appreciate the efforts that my advisor, the department of Aeronautics and Astronautics, and MIT have made to facilitate my work during these months. I would like to sincerely thank my advisor, Prof. Edward Crawley, and Dr. Bruce Cameron for their support, advice, and guidance throughout these past two years. Ed, Bruce, I really appreciate your honest input and implication to make this research project succeed. It is truly an honor to work side by side with you. I would also like to thank everyone else that has been part of this research project. First, I really appreciate the support received from SES, especially the feedback and motivation from Joel Grotz and Valvanera Moreno. Next, I would like to express my gratitude to my labmate Markus Guerster, with whom I have shared many moments of joy during the project. Markus, working together on this project has been a rewarding learning experience, I sincerely wish you all the best in your future endeavors. I also appreciate the valuable inputs and help received from Dr. Kalyan Veeramachaneni. I would like to acknowledge the rest of my labmates who also have been part of this project at some point during these two years. Nils, Damon, Rubén, and Skylar, you have been a source of inspiration and stimulating discussions, best of luck on your next steps. The rest of my labmates at the Engineering Systems Lab have been a key factor to succeed in this first part of my graduate studies. Sydney, Matt, Alex, Anne, Eric, Beldon, George, Tommy, Katie, Michael, thank you for your constant encouragement and for all the fun times together. I would also like to show my appreciation to my former labmate, Íñigo del Portillo, for his valuable advice and honest guidance during my first year at the lab. I do not want to miss the chance to say thank youforthe invaluable administrative support I always get from Amy Jarvis, Beth Marois, and 5 Ping Lee, and the counseling received from Suraiya Baluch. Going through these two years would not have been possible without the support of my roomate Marc de Cea. Marc, I feel very fortunate to share the MIT adventure with you. I am also deeply grateful to my friends for their continous doses of joy and fun throughout these two years: María, Inés, Helena, Ximo, Alex, Dani, Álvaro, Reus, Íñigo, Ondrej, Lukas, Regina, Faisal. Also, being part of Spain@MIT has been so much fun! I also want to thank the tremendous support received from my family and friends in Spain. Big thanks to my parents, Ana and Simón, for encouraging me to work hard and aim high. Also, thanks to my grandparents, godparents, aunts, uncles, cousins and the rest of my family for motivating me and constantly checking in. I am also deeply grateful to my closest friends – you know who you are – who always show me that distance is nothing when it comes to our friendship. Finally, I would like to dedicate this Thesis to my late grandmother Antònia, who passed away shortly before it was completed. Thank you for everything you have done for me, pradina. 6 Contents 1 Introduction 19 1.1 Motivation . 19 1.2 General Objectives . 22 1.3 Literature Review . 23 1.4 Specific Objectives . 26 1.5 Thesis Overview . 28 2 Dynamic Power Allocation in Multibeam Satellites 31 2.1 Introduction . 31 2.2 Dynamic Resource Management . 32 2.3 Multibeam Satellite Communications Systems . 33 2.3.1 Overview . 33 2.3.2 Dynamic Resource Management in Multibeam Satellites . 35 2.3.3 Artificial Intelligence for the DRM problem in satellite commu- nications . 38 2.4 Dynamic Power Allocation Problem . 40 2.4.1 Power Allocation and Transmission . 41 2.4.2 Problem Statement . 43 2.4.3 Objective Metrics . 45 3 Algorithm Implementations 47 3.1 Introduction . 47 3.2 Metaheuristics . 47 7 3.2.1 Genetic Algorithm . 48 3.2.2 Simulated Annealing . 49 3.2.3 Particle Swarm Optimization . 52 3.3 Deep Reinforcement Learning . 53 3.4 Hybrid Algorithms . 58 3.4.1 SA-GA Hybrid . 58 3.4.2 PSO-GA Hybrid . 59 3.4.3 DRL-GA Hybrid . 60 4 Simulation Models 61 4.1 Introduction . 61 4.2 Satellite Model . 61 4.3 Demand Models . 62 4.4 Link Budget Model . 64 5 Results 67 5.1 Introduction . 67 5.2 Convergence Analyses . 68 5.3 Continuous Operation Performance . 71 5.4 Scalability Analyses . 77 5.5 Robustness Analyses . 81 5.5.1 Sequential Activation . 82 5.5.2 Spurious Events . 85 5.5.3 Non-stationarity . 88 5.5.4 Conclusions on robustness . 92 6 Conclusions 95 6.1 Thesis Summary . 95 6.2 Main Findings . 97 6.3 Future Work . 99 8 A Additional Figures 101 A.1 Convergence Analyses . 101 A.2 Continuous Operation . 102 A.3 Scalability Analyses . 104 A.4 Robustness Analyses . 109 B Metric details 113 B.1 Satisfaction-Gap Measure . 113 B.2 Power calculation heuristic . 114 9 10 List of Figures 1-1 Data rate provided by a slightly flexible system (blue) and a highly flexible system (green) with respect to the requested data rate (red). The amount of resource savings corresponds to the area between the green and the blue curves. ...... 20 2-1 Multibeam satellite with 7 beams. ................. 34 3-1 DRL Architecture. ........................... 55 4-1 Normalized aggregated demand plot for the four scenarios considered. ................................ 64 5-1 Average aggregated power and 95% confidence interval against computing time available. Power is normalized with respect to the optimal aggregated power. Reference scenario used. 70 5-2 Average aggregated UD and 95% confidence interval against computing time available. UD is normalized with respect to the aggregated demand. Reference scenario used. ....... 70 5-3 Aggregated power delivered by every algorithm during the continuous execution simulations. Power is normalized with respect to the optimal aggregated power (optimal power is 1). Reference scenario used. ....................... 72 11 5-4 Aggregated UD achieved by every algorithm during the con- tinuous execution simulations. UD is normalized with respect to aggregated demand (optimal UD is 0).