Program Synthesis with Grammars and Semantics in Genetic Programming Stefan Forstenlechner UCD student number: 14204817 The thesis is submitted to University College Dublin in fulfilment of the requirements for the degree of DOCTOR OF PHILOSOPHY School of Business Head of School: Prof. Anthony Brabazon Research Supervisors: Prof. Michael O’Neill Dr. Miguel Nicolau External Examiner: Dr. John R. Woodward January 2019 Contents Contents i Abstract vii Statement of Original Authorship viii Acknowledgements ix List of Figures xi List of Tables xiv List of Algorithms xvii List of Abbreviations xviii Publications Arising xx I Introduction and Literature Review 1 1 Introduction 2 1.1 Aim of Thesis . .3 1.2 Research Questions . .4 1.3 Contributions . .7 1.3.1 Technical contributions . .8 1.4 Limitations . .8 1.5 Thesis Outline . .9 i CONTENTS 2 Related Work 12 2.1 Evolutionary Computation . 12 2.2 Genetic Programming . 14 2.2.1 Representation . 15 2.2.2 Initialization . 16 2.2.3 Fitness . 17 2.2.4 Selection . 17 2.2.5 Crossover . 19 2.2.6 Mutation . 19 2.2.7 GP Summary . 20 2.2.8 Grammars . 20 2.3 Program Synthesis . 23 2.3.1 Program Synthesis in Genetic Programming . 26 2.3.2 General Program Synthesis Benchmark Suite . 30 2.4 Semantics . 32 2.4.1 Semantic operators . 34 2.4.2 Geometric Semantic GP . 37 2.5 Conclusion . 38 II Experimental Research 40 3 General Grammar Design 41 3.1 Grammars for G3P . 41 3.2 Sorting Network . 42 3.3 Structure in Grammars . 43 3.4 Sorting Network Grammar Design . 45 3.4.1 Derivation tree sizes . 49 3.4.2 Grammar Design Details . 50 3.5 Experimental Setup . 51 3.5.1 Experiment 1 . 51 3.5.2 Experiment 2 . 51 3.5.3 General Settings . 52 3.6 Results . 53 3.6.1 Experiment 1 . 53 3.6.2 Experiment 2 . 54 ii CONTENTS 3.7 Summary . 58 4 Program Synthesis Grammar Design Pattern 60 4.1 Previous Approaches to Program Synthesis . 60 4.1.1 Grammar-Guided Genetic Programming . 61 4.1.2 Strongly Formed Genetic Programming . 61 4.1.3 PushGP . 62 4.1.4 Summary . 63 4.2 System Description . 63 4.2.1 Grammar Design Pattern . 64 4.2.2 Skeleton . 67 4.2.3 Comparison of Program Synthesis Approaches . 68 4.2.4 Python Specific Differences . 70 4.2.5 Invalid Individuals . 71 4.3 Experimental Setup . 72 4.3.1 PushGP Differences . 73 4.3.2 Derivation Tree Structures . 74 4.4 Results . 75 4.4.1 Tournament Selection . 75 4.4.2 Lexicase Selection . 76 4.4.3 Generational Progress and Invalids . 79 4.4.4 Derivation Tree Structures . 81 4.5 Summary . 81 III Extended Experimental Research 84 5 Refining Computational Effort 85 5.1 General Program Synthesis Benchmark Suite Remarks . 86 5.2 Experimental Setup . 87 5.2.1 Parameters and Computational Effort . 87 5.2.2 Larger Training Set . 89 5.3 Results . 90 5.3.1 Success Rates . 90 5.3.2 Accumulated Successful Solutions Over Generations . 92 5.3.3 Problems with the Training Data . 94 iii CONTENTS 5.3.4 Larger Training Set . 95 5.4 Computational Effort Discussion . 97 5.5 Benchmark Suite Discussion . 98 5.6 Summary . 99 6 Extending Program Synthesis Grammars 101 6.1 Grammar Design Approach Remarks . 101 6.2 Extending Program Synthesis Grammars . 103 6.2.1 Data Type Char . 103 6.2.2 Recursion . 104 6.2.3 List Operations . 105 6.2.4 Additional Methods . 106 6.3 Experimental Setup . 106 6.4 Results . 107 6.4.1 Successful Solutions . 107 6.4.2 Char Analysis . 108 6.4.3 Recursion Analysis . 111 6.5 Summary . 111 7 Semantic Operators in Program Synthesis 114 7.1 Semantics . 115 7.1.1 Semantic Operators . 115 7.2 Semantics in Program Synthesis . 116 7.3 Semantic Crossover for Program Synthesis . 120 7.3.1 Semantic Measure . 120 7.3.2 Operator . 121 7.3.3 Experimental Setup . 124 7.3.4 Results . 125 7.3.5 Summary of SCPS . 132 7.4 Effective Semantic Operators for Program Synthesis . 134 7.4.1 Effective Semantic Crossover for Program Synthesis . 134 7.4.2 Effective Semantic Mutation for Program Synthesis . 136 7.4.3 Experimental Setup . 137 7.4.4 Results . 137 iv CONTENTS 7.4.5 Summary of Effective Semantic Operators for Program Syn- thesis . 146 7.5 Summary . 146 IV Fin. 148 8 Conclusion & Future Work 149 8.1 Thesis Summary . 149 8.2 Contributions . 151 8.2.1 Technical contributions . 152 8.3 Limitations . 153 8.4 Future Work . 154 A Program Synthesis Problem Description 156 A.1 Problem Description . 156 A.2 Fitness Functions . 159 B Program Synthesis Grammars 162 B.1 Automatic Grammar Combination . 162 B.2 structure.bnf . 164 B.3 bool.bnf . 165 B.4 float.bnf . 165 B.5 int.bnf . 166 B.6 string.bnf . 167 B.7 list_bool.bnf . 168 B.8 list_float.bnf . 169 B.9 list_int.bnf . 170 B.10 list_string.bnf . 171 B.11 Protected methods . 173 C Extended Program Synthesis Grammars 176 C.1 structure.bnf . 176 C.2 bool.bnf . 177 C.3 float.bnf . 178 C.4 int.bnf . 179 C.5 char.bnf . 180 v CONTENTS C.6 string.bnf . 181 C.7 list_bool.bnf . 183 C.8 list_float.bnf . ..