Building Compiled Language Extensions for Javascript
Total Page:16
File Type:pdf, Size:1020Kb
Bachelor Thesis Building Compiled Language Extensions for JavaScript Author Examiners Jonathan Immanuel Brachthäuser Prof. Walter Kriha Prof. Dr. Ansgar Gerlicher Andreas Stiegler, M. Sc. Stuttgart Media University (HdM) — February 27, 2012 Integrity Statement Herein I declare that this Bachelor Thesis was created entirely by myself. I only used the sources specifically stated in this document. Thoughts used, either by meaning or quoted, were marked as such. Stuttgart, February 27, 2012 Jonathan Immanuel Brachthäuser iii Acknowledgements This thesis would be impossible without the support of some great people. Marlen, I love you. Thank you for trying to understand what I am doing and especially for learning Lisp! Thank you, beloved family, for always supporting me unconditionally. Many thanks to Stephan Soller, Andreas Stiegler and Patrick Bader for all those intense and vivid discussions that helped my stay on track. I have truly learned a lot from you guys, even in times I did not want to. Patrick pointed my nose at OMeta which has been a great inspiration for this thesis. By the way, AST guessing is an awesome game to play. Alessandro Warth did a great work with his language OMeta. Thank you, it is so much fun to work with. Even if writing a thesis can be rough some time it is less of a burden when working in a comforting environment. Thank you Prof. Zimmermann for your trust and for tilting at the windmills of bureaucracy. I also have to thank Claus Gittinger for showing me that building Interpreters (and compilers in extension) is no voodoo. Last but not least I want to thank Prof. Walter Kriha and Prof. Dr. Ansgar Gerlicher for giving me the chance and freedom to do research on such an amazing topic. v Abstract The web is moving in an enormous speed. New standards like HTML5, CSS3 and the 6th edition of the ECMAScript are under development. Browser vendors have improved the performance of their JavaScript engines a lot which now serve as Petri dishes for the development of new languages and JavaScript derivatives. The term “transpiler” has been coined to subsume the cross-compilation from one edition of ECMAScript to another. Set in this volatile environment, the goal of this thesis is to provide a foundation to compose individual extensions. Thus, a custom language derivative can be built which fits the individual likings. It is not scope of this thesis to design such a language but to provide a framework supporting the creation of JavaScript language extensions. This thesis targets at intermediate JavaScript developers who are interested in creating their own language. No expert knowledge in building compilers is required. The thesis is designed as a guide and hence necessary theoretical knowledge of compilers and parser generators is briefly discussed in the first few chapters. OMeta/JS is chosen as parser generator in oder to experiment with language extensions in the most flexible and convenient way. Both a JavaScript parser and a translator are implemented in a way that allows an easy extension. JsonML is chosen to internally represent the abstract syntax tree. The creation of the tree is accomplished by the means of node-constructors. A generic walker grammar is provided in order to easily traverse the syntax tree. The design of the framework aims at the reuse of the single components. Hence, the re-factored implementation of OMeta/JS can be used as a standalone package for Node.js. The interdependencies between the different modules are clearly separated to allow reasonable maintainability. A fictional language EJS is introduced to demonstrate the possibilities of the framework at hand. For this purpose four example extensions are discussed, each with increasing complexity. The last extension finally introduces a class based object orientation. Many improvements of the framework are imaginable. Yet, the architecture renders itself to be a good foundation to quickly develop JavaScript language extensions. vii Contents Page 1 Introduction 1 1.1 JavaScript . 2 1.1.1 Outside of the Browser . 2 1.1.2 JavaScript as a Standard . 3 1.1.3 Problems and Solutions . 4 1.1.4 Compiled Language Extensions . 5 1.2 Scope of this Thesis . 6 2 The Compilation Process 9 2.1 Lexical Analysis . 10 2.2 Syntactical Analysis . 11 2.3 Tree Transformations . 12 2.4 Code Generation . 13 2.5 Summary . 13 3 A little more about Parsers 15 3.1 Recursive Decent Parsers . 16 3.2 Extending Recursive Decent Parsers . 19 3.2.1 Backtracking . 19 3.2.2 Memoization . 20 3.2.3 Left Recursion . 21 3.2.4 Parsing Expression Grammars . 22 3.2.5 Semantic Predicates . 23 3.3 Summary . 23 ix CONTENTS 4 Existing Parser Generators 25 4.1 Comparison of PEG-Parser Generators . 26 4.1.1 Concepts . 27 4.1.2 Documentation . 27 4.1.3 Error Reporting . 28 4.1.4 Extensibility . 28 4.1.5 Conclusion . 29 5 OMeta 31 5.1 Writing Grammars . 33 5.1.1 Differences to PEG . 33 5.1.2 Pattern Matching . 34 5.1.3 Semantic Predicates . 35 5.1.4 Semantic Actions . 36 5.1.5 Parametrized Rules . 38 5.1.6 Higher-Order Rules . 39 5.1.7 It’s all about Context: OMeta or JavaScript? . 40 5.1.8 Grammar Inheritance . 41 5.1.9 Foreign Rule Invocation . 42 5.2 Using OMeta/JS . 43 5.2.1 Usage of OMeta Grammar Objects . 44 5.2.2 Stateful Pattern Matching . 45 5.3 Summary . 46 6 Extending JavaScript 47 6.1 Five Steps to Create a Language Extension . 48 6.1.1 First Step: Set up the Environment . 49 6.1.2 Second Step: Write an ES5 Grammar . 51 6.1.3 Third Step: Specify the Format of the AST . 53 6.1.4 Fourth Step: Traverse the AST and Generate Code . 56 6.1.5 Fifth Step: Start Extending . 61 6.2 The Architecture . 64 6.2.1 The OMeta Package . 65 6.2.2 The JsonML Package . 66 6.2.3 The ES5 Package . 67 6.2.4 The EJS Package . 70 6.3 The Grammars . 72 6.4 Summary . 74 x CONTENTS 7 Usecase: Example Extension 75 7.1 Scope Forcing with !{} ............................. 76 7.1.1 Introducing a new Syntax . 78 7.1.2 Implementing the Syntax . 79 7.2 The Substitution Operator “#{}“ ........................ 80 7.2.1 Introducing a new Syntax . 81 7.2.2 Implementing the Syntax . 81 7.3 Lambda Expressions {||} ............................ 82 7.3.1 Introducing a new Syntax . 83 7.3.2 Implementing the Syntax . 85 7.4 Classes and Object Orientation . 86 7.4.1 Introducing a new Syntax . 88 7.4.2 Implementing the Syntax . 90 7.5 Summary . 93 8 Conclusions 95 8.1 Related Work . 95 8.2 Future Work . 96 A Code Samples 99 A.1 PEG-Grammars . 100 A.2 Visitor Based Implementation . 102 A.3 Recursive Descent Lisp Parser . 103 A.4 Implementation of Implicit Returns . 104 B OMeta/JS 105 B.1 OMeta/JS - Required Files for Compilation . 106 B.2 OMeta Base Grammar . 107 B.3 Object Pattern Matching in OMeta/JS . 108 Bibliography 110 List of Figures 118 List of Tables 119 Abbreviations 119 xi Chapter 1 Introduction Contents 1.1 JavaScript . 2 1.1.1 Outside of the Browser . 2 1.1.2 JavaScript as a Standard . 3 1.1.3 Problems and Solutions . 4 1.1.4 Compiled Language Extensions . 5 1.2 Scope of this Thesis . 6 For almost every developer there are times, when the question arises: “Why, tell me why, can’t I simply do it like in x?” In this saying, x can be replaced by any programming language that might have one additional feature to that language one is currently working with. This absent feature can be a complex one, unique to the language. However, these features often are of syntactical nature - in other words: just the way to express the things we want to happen. With every new programming language we have learned, we also possibly gain more frustration because there is no programming language that can fit all shoes. So most of the time we will be missing certain capabilities or ways to express our algorithms in a delightful manner. There are languages that make us feel more locked-in than others do. On the other hand, some existing languages are full of beautiful features, waiting to be discovered1. Additionally there is a third category of languages, providing us with all the necessary tools to change the language itself where it doesn’t fit our needs. A good example for this group is Lisp, where every programmer can create his own set of tools and redefine almost anything, according to his personal likings. Nevertheless, thsese shortcomings usually exist for a purpose. Sometimes “prosaic elegance” has been traded off against performance, security motives or internal consis- tency. So there are many good reasons that underly possible answers to the introductory question. Likely none of them will really comfort us in that kind of situation. We just have to keep in mind that languages are not born out of a singularity, somewhere in vacuo. Instead, languages are settled in a given environment as children of hard requirements 1According to my personal experiences, Ruby is a good example for this languages 1 CHAPTER 1. INTRODUCTION that have a large impact on design decisions. Occasionally, mostly due to improvements of the underlying hardware, some of these requirements become obsolete with time passing, but others still apply. 1.1 JavaScript JavaScript has been developed as language within the browser. With the magnificent gain of importance the Internet has received in the last decade, JavaScript has become one of the world’s most used programming languages.