Part II Parsing and Disambiguation of Source Code
Total Page:16
File Type:pdf, Size:1020Kb
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/238758491 Analysis and Transformation of Source Code by Parsing and Rewriting Article · January 2005 CITATIONS READS 43 305 1 author: Jurgen Vinju Centrum Wiskunde & Informatica 145 PUBLICATIONS 1,926 CITATIONS SEE PROFILE Some of the authors of this publication are also working on these related projects: CROSSMINER: Developer-Centric Knowledge Mining from Large Open-Source Software Repositories View project ASF+SDF Meta-Environment View project All content following this page was uploaded by Jurgen Vinju on 10 January 2014. The user has requested enhancement of the downloaded file. Downloaded from UvA-DARE, the institutional repository of the University of Amsterdam (UvA) http://dare.uva.nl/document/17005 File ID 17005 SOURCE (OR PART OF THE FOLLOWING SOURCE): Type Dissertation Title Analysis and transformation of source code by parsing and rewriting Author J.J. Vinju Faculty Faculty of Science Year 2005 Pages 230 FULL BIBLIOGRAPHIC DETAILS: http://dare.uva.nl/record/165444 Copyright It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other then for strictly personal, individual use. UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl) Analysis and Transformation of Source Code by Parsing and Rewriting ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit van Amsterdam op gezag van de Rector Magnificus prof. mr. P.F. van der Heijden ten overstaan van een door het college voor promoties ingestelde commissie, in het openbaar te verdedigen in de Aula der Universiteit op dinsdag 15 november 2005, te 10:00 uur door Jurgen Jordanus Vinju geboren te Ermelo Promotor: prof. dr. P. Klint Co-promotor: dr. M.G.J. van den Brand Faculteit: Natuurwetenschappen, Wiskunde en Informatica The work in this thesis has been carried out at Centrum voor Wiskunde en Informat- ica (CWI) in Amsterdam under the auspices of the research school IPA (Institute for Programming research and Algorithmics). Preface Before consuming this manuscript the reader should know that I owe gratitude to many people. First of all, it is a family accomplishment. I want to thank my mother, Annelies, for being so strong and for always motivating me to do what I like best. I thank my father, Fred, for always supporting me. My sister, Krista, is my soul mate. We are so much alike. The love of my live, Rebecca, has been my support and inspiration for the past six years. I would like to thank my best friends: Arjen Koppen, Bas Toeter, Coen Visser, Hugo Loomans, Mieke Schouten, Warner Salomons, and Winfried Holthuizen. You don't know how much of you is a part of me. My supervisors at CWI are Mark van den Brand and Paul Klint. Mark inspired me to study computer science in Amsterdam, and to start a PhD project at CWI. He sparked my interest in ASF+SDF already at the age of 17. Thank you for teaching me, caring for me, and for motivating me all these years. Paul is a great mentor and role model. I admire him for his insight in so many issues and for his endless enthusiasm for research. He is the most productive man I have ever seen. Thanks for your time teaching me. Many of my colleagues have become my friends. Thank you for the teamwork, for providing an inspiring work environment, and for the relaxing times we spent in bars and restaurants. They are in alphabetical order: Ali Mesbah, Anamaria Martins Mor- eira, Anderson Santana, Anthony Cleve, Arie van Deursen, Li Bixin, Chris Verhoef, David Deharbe,´ Diego Ordonez Camacho, Eelco Visser, Ernst-Jan Verhoeven, Gerald Stap, Gerco Ballintijn, Hayco de Jong, Jan Heering, Jan van Eijck, Jeroen Scheerder, Joost Visser, Jørgen Iversen, Steven Klusener, Leon Moonen, Magiel Bruntink, Martin Bravenboer, Merijn de Jonge, Niels Veerman, Pierre-Etienne Moreau, Pieter Olivier, Ralf Lammel,¨ Rob Economopoulos, Slinger Jansen, Taeke Kooiker, Tijs van der Storm, Tobias Kuipers, Tom Tourwe,´ Vania Marangozova. I would like to thank Claude Kirchner for allowing me to work an inspiring and productive three month period at INRIA-LORIA. Finally, I thank the members of my reading committee for reading the manuscript and providing valuable feedback: prof. dr. J.A. Bergstra, prof. dr. M. de Rijke, prof. dr. C.R. Jesshope, prof. dr. K.M. van Hee and prof. dr. J.R. Cordy. The CWI institute is a wonderful place to learn and produce computer science. v vi Contents Contents vii I Overview 1 1 Introduction 3 1.1 Computer aided software engineering . 3 1.1.1 Source code . 4 1.1.2 Source code analysis and transformation . 6 1.1.3 Translation distance . 6 1.1.4 Goals and requirements . 7 1.1.5 Mechanics . 8 1.1.6 Discussion: challenges in meta programming . 9 1.2 Technological background . 10 1.2.1 Generic language technology . 10 1.2.2 A meta-programming framework . 11 1.2.3 Historical perspective . 12 1.2.4 Goal . 13 1.3 Parsing . 14 1.3.1 Mechanics . 14 1.3.2 Formalism . 14 1.3.3 Technology . 15 1.3.4 Application to meta-programming . 16 1.4 Rewriting . 17 1.4.1 Mechanics . 17 1.4.2 Formalism . 18 1.4.3 Technology . 19 1.4.4 Application to meta-programming . 20 1.5 Related work . 23 1.6 Road-map and acknowledgments . 24 vii 2 Environments for Term Rewriting Engines for Free! 29 2.1 Introduction . 29 2.2 Architecture for an open environment . 31 2.3 Reusable components . 33 2.3.1 Generalized Parsing for a readable formalism . 33 2.3.2 Establishing the connection between parsing and rewriting . 34 2.3.3 Graphical User Interface . 35 2.4 A new environment in a few steps . 36 2.5 Instantiations of the Meta-Environment . 40 2.6 Conclusions . 41 II Parsing and disambiguation of source code 43 3 Disambiguation Filters for Scannerless Generalized LR Parsers 45 3.1 Introduction . 45 3.2 Scannerless Generalized Parsing . 46 3.2.1 Generalized Parsing . 46 3.2.2 Scannerless Parsing . 47 3.2.3 Combining Scannerless Parsing and Generalized Parsing . 48 3.3 Disambiguation Rules . 49 3.3.1 Follow Restrictions . 49 3.3.2 Reject Productions . 50 3.3.3 Priority and Associativity . 50 3.3.4 Preference Attributes . 51 3.4 Implementation Issues . 52 3.4.1 Follow Restrictions . 52 3.4.2 Reject Productions . 53 3.4.3 Priority and Associativity . 53 3.4.4 Preference Attributes . 54 3.5 Applications . 55 3.5.1 ASF+SDF Meta-Environment . 55 3.5.2 XT . 55 3.6 Benchmarks . 56 3.7 Discussion . 57 3.7.1 Generalized LR parsing versus backtracking parsers . 57 3.7.2 When to use scannerless parsing? . 57 3.8 Conclusions . 58 4 Semantics Driven Disambiguation 59 4.1 Introduction . 59 4.1.1 Examples . 60 4.1.2 Related work on filtering . 62 4.1.3 Filtering using term rewriting . 63 4.1.4 Plan of the chapter . 63 4.2 Parse Forest Representation . 64 viii 4.3 Extending Term Rewriting . 65 4.3.1 What is term rewriting? . 66 4.3.2 Rewriting parse trees . 68 4.3.3 Rewriting parse forests . 69 4.4 Practical Experiences . 70 4.5 Discussion . 72 4.6 Conclusions . 73 5 A Type-driven Approach to Concrete Meta Programming 75 5.1 Introduction . 75 5.1.1 Exploring the solution space . 77 5.1.2 Concrete meta programming systems . 79 5.1.3 Discussion . 83 5.2 Architecture . 84 5.2.1 Syntax transitions . 85 5.2.2 Disambiguation by type-checking . 87 5.3 Disambiguation filters . 88 5.3.1 Class 3. Ambiguity directly via syntax transitions . 88 5.3.2 Class 4. Object language and meta language overlap . 92 5.4 Experience . 94 5.5 Conclusion . 95 III Rewriting source code 97 6 Term Rewriting with Traversal Functions 99 6.1 Introduction . 99 6.1.1 Background . 99 6.1.2 Plan of the Paper . 100 6.1.3 Issues in Tree Traversal . 100 6.1.4 A Brief Recapitulation of Term Rewriting . 102 6.1.5 Why Traversal Functions in Term Rewriting? . 104 6.1.6 Extending Term Rewriting with Traversal Functions . 106 6.1.7 Related Work . 108 6.2 Traversal Functions in ASF+SDF . 111 6.2.1 Kinds of Traversal Functions . 112 6.2.2 Visiting Strategies . 113 6.2.3 Examples of Transformers . 114 6.2.4 Examples of Accumulators . 117 6.2.5 Examples of Accumulating Transformers . 118 6.3 Larger Examples . 119 6.3.1 Type-checking . 119 6.3.2 Inferring Variable Usage . 124 6.3.3 Examples of Accumulating Transformers . 124 6.4 Operational Semantics . 125 6.4.1 Extending Innermost . 126 ix 6.4.2 Transformer . 127 6.4.3 Accumulator . 127 6.4.4 Accumulating Transformer . 128 6.4.5 Discussion . 128 6.5 Implementation Issues . ..