Automatic Non-Functional Testing and Tuning of Configurable Generators Mohamed Boussaa
Total Page:16
File Type:pdf, Size:1020Kb
Automatic non-functional testing and tuning of configurable generators Mohamed Boussaa To cite this version: Mohamed Boussaa. Automatic non-functional testing and tuning of configurable generators. Math- ematical Software [cs.MS]. Université Rennes 1, 2017. English. NNT : 2017REN1S011. tel- 01598821v2 HAL Id: tel-01598821 https://tel.archives-ouvertes.fr/tel-01598821v2 Submitted on 19 Oct 2017 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. ANNÉE 2017 THÈSE / UNIVERSITÉ DE RENNES 1 sous le sceau de l’Université Bretagne Loire pour le grade de DOCTEUR DE L’UNIVERSITÉ DE RENNES 1 Mention : Informatique École doctorale Matisse présentée par Mohamed BOUSSAA préparée à l’unité de recherche INRIA INRIA Rennes Bretagne Atlantique ISTIC Thèse soutenue à Rennes le 6 Septembre 2017 Automatic devant le jury composé de : Non-functional Hélène WAESELYNCK Directrice de recherche, LAAS-CNRS Toulouse / Testing and Tuning Rapporteuse Philippe MERLE Chargé de recherche, INRIA Lille / Rapporteur of Configurable Erven ROHOU Directeur de recherche, INRIA Rennes / Examinateur Generators Franck FLEUREY Directeur de recherche, SINTEF Oslo / Examinateur Jean-Marie MOTTU Maître de conférences, Université de Nantes / Examinateur Gerson SUNYÉ Maître de conférences, Université de Nantes / Examinateur Benoit BAUDRY Directeur de recherche, INRIA Rennes / Directeur de thèse Olivier BARAIS Professeur, Université de Rennes 1 / Co-directeur de thèse Acknowledgements This thesis would not have been completed without the help of others. I would like to take this opportunity to express my gratitude towards them and acknowledge them. First of all, I would like to offer my deepest gratitude to my supervisors Olivier Barais and Benoit Baudry for their useful advices, and for guiding my steps over more than three years. Special thanks are also due to my \third" supervisor Gesron Suny´e that always knew how to motivate me and help me to overcome the difficult times in my thesis. I owe special thanks to my Master's supervisor Marouane Kessentini, with whom I discovered the software engineering research world, for encouraging me to continue in the research field as a PhD candidate. I would also like to thank the members of my jury. My examiners, H´el`eneWaeselynck and Philippe Merle for having accepted to serve on my examination board and for the time they have invested in reading and evaluating this thesis. Thanks are also due to Erven Rohou for agreeing to preside over my jury. I would also like to thank Jean-Marie Mottou and Franck Fleury for attending as members of the jury. I greatly appreciate and wish to thank all the (former and current) members of the DiverSE research team for providing a great and friendly work atmosphere, and for all the different discussions, seminars, and DiverSE coffees. Finally, I would like to thank my family. Even so far away from home, they are always present in my life during these three years. I first express my gratitude to my parents Najoua and Noureddine for their support, comprehension, confidence and, especially for their patience. You have always been behind me and believed in me. Then, thanks to my sisters Imene and Jihene for encouraging me to succeed in everything I do. By now, we can say that we have a Doctor in the Boussaa family ! i Contents R´esum´een Fran¸cais vii Contexte........................................ vii Motivations...................................... vii Probl´ematiques.................................... ix Contributions..................................... x 1 Introduction1 1.1 Context ..................................... 1 1.2 Motivation .................................... 2 1.3 Scope of the thesis ............................... 4 1.4 Challenges .................................... 5 1.5 Contributions .................................. 6 1.6 Overview of this thesis ............................. 8 1.7 Publications ................................... 9 I Background and State of the Art 11 2 Background 13 2.1 Diversity in software engineering ........................ 14 2.1.1 Hardware heterogeneity ........................ 14 iii iv CONTENTS 2.1.2 Software diversity............................ 16 2.1.3 Matching software diversity to heterogeneous hardware: the marriage 18 2.2 From classical software development to generative programming . 22 2.2.1 Overview of the generative software development process . 23 2.2.2 Automatic code generation in GP: a highly configurable process . 26 2.2.3 Stakeholders and their roles for testing and tuning generators . 28 2.3 Testing code generators ............................. 29 2.3.1 Testing workflow ............................ 30 2.3.2 Types of code generators ........................ 30 2.3.3 Why testing code generators is complex? . 33 2.4 Compilers auto-tuning ............................. 34 2.4.1 Code optimization ........................... 35 2.4.2 Why compilers auto-tuning is complex? . 36 2.5 Summary: challenges for testing and tuning configurable generators . 37 3 State of the art 39 3.1 Testing code generators ............................. 40 3.1.1 Functional testing of code generators . 40 3.1.2 Non-functional testing of code generators . 46 3.2 Compilers auto-tuning techniques ....................... 51 3.2.1 Iterative compilation .......................... 51 3.2.2 Implementation of the iterative compilation process . 51 3.2.3 Iterative compilation search techniques . 53 3.3 Lightweight system virtualization for automatic software testing . 61 3.3.1 Application in software testing ..................... 62 3.3.2 Application in runtime monitoring . 64 3.4 Summary & open challenges .......................... 65 v II Contributions 67 To the reader: summary of contributions 69 4 Automatic non-functional testing of code generator families 73 4.1 Context and motivations............................ 74 4.1.1 Code generator families ......................... 74 4.1.2 Issues when testing a code generator family . 76 4.2 The traditional process for non-functional testing of a code generator family 76 4.3 Approach overview ............................... 78 4.3.1 An infrastructure for non-functional testing using system containers 78 4.3.2 A metamorphic testing approach for automatic detection of code generator inconsistencies ........................ 80 4.4 Evaluation .................................... 89 4.4.1 Experimental setup ........................... 90 4.4.2 Experimental methodology and results . 93 4.4.3 Threats to validity . 102 4.5 Conclusion .................................... 103 5 NOTICE: An approach for auto-tuning compilers 105 5.1 Motivation .................................... 107 5.2 Evolutionary exploration of compiler optimizations . 110 5.2.1 Novelty search adaptation . 110 5.2.2 Novelty search for multi-objective optimization . 113 5.3 Evaluation .................................... 114 5.3.1 Research questions . 114 5.3.2 Experimental setup . 115 5.3.3 Experimental methodology and results . 118 5.3.4 Discussions ............................... 127 vi CONTENTS 5.3.5 Threats to validity ........................... 128 5.3.6 Tool support overview ......................... 128 5.4 Conclusion.................................... 131 6 A lightweight execution environment for automatic generators testing 133 6.1 Introduction ................................... 133 6.2 System containers as a lightweight execution environment . 134 6.3 Runtime Monitoring Engine . 136 6.3.1 Monitoring Container . 136 6.3.2 Back-end Database Container . 137 6.3.3 Front-end Visualization Container . 138 6.4 The generator case study ............................ 138 6.5 Conclusion .................................... 141 III Conclusion and Perspectives 143 7 Conclusion and perspectives 145 7.1 Summary of contributions . 145 7.2 Perspectives ................................... 147 References 151 List of Figures 171 List of Tables 173 R´esum´een Fran¸cais Contexte Les techniques ´emergentes de l'ing´enierie dirig´eepar les mod`eleset de la programma- tion g´en´erative ont permis la cr´eationde plusieurs g´en´erateurs(g´en´erateurs de code et compilateurs). Ceux-ci sont souvent utilis´esafin de faciliter le d´eveloppement logiciel et automatiser le processus de g´en´erationde code `apartir des sp´ecificationsabstraites. En effet, les utilisateurs sont devenus capables de synth´etiserrapidement des artefacts logiciels pour une large gamme de plateformes logicielles et mat´erielles. En outre, les g´en´erateurs modernes comme les compilateurs C, sont devenus hautement configurables, offrant de nombreuses options de configuration `al'utilisateur de mani`ere`apersonnaliser facilement le code g´en´er´epour la plateforme mat´eriellecible. Par cons´equent, la qualit´elogicielle est devenue fortement corr´el´eeaux param`etresde configuration ainsi qu'au g´en´erateurlui- m^eme.Dans ce contexte, il est devenu indispensable de v´erifierle bon comportement des g´en´erateurs.D'une part, il est important de tester les programmes g´en´er´esafin de d´etecter des anomalies de g´en´erateurset les corriger. D'autre