Imperial College of London Department of Computing JErlang: Erlang with Joins Final Report Hubert Plociniczak Supervisor Professor Susan Eisenbach Second marker Professor Sophia Drossopoulou Copyright c Hubert Plociniczak, 2009. All rights reserved. Abstract For the past few years the computing world has been affected by a massive increase in the popularity of multi-core machines. In order to fully utilise this opportunity, programmers have to switch their mindset to concurrent thinking. The human nature allows for quick adaptation to new environment, yet ironically, the tools that we use to program the concur- rent programs are still in the late 80's where sequential programming was rampant. Writing massive concurrent applications is an inherently error prone process due to the way humans think. Nevertheless popular languages like Java or C# still use locking as a basic concurrent abstraction. Erlang, a precursor in the world of concurrent languages, offers message passing instead of shared memory, as a way of avoiding this issue. Yet this idea itself is also very primitive and the whole burden of managing large amounts of processes that use the possibilities of multi-core machines, is again left on the shoulders of programmers. This report investigates the development of JErlang, an extension of Erlang, that in- troduces simple yet powerful joins constructs. The idea is adapted from the Join-calculus, a process calculus that has recently been developed by Fournet and Gonthier. The aim of JErlang is to allow the programmer for easier synchronisation between different processes in an intuitive way. This will enable Erlang programmers to write concurrent applications faster, reliably and increase the overall clarity of the programs. Acknowledgements This project would not reach this stage without the help of many people, a few of which I will name below. I would like to thank Professor Susan Eisenbach, for her continuous support and encouragement not only during this project but during the whole duration of my study. Secondly, I would like to thank Natalia, for her patience during the nervous times and smile that helped me to survive each day. Many thanks also go to Professor Sophia Drossopoulou who helped me in thinking for- mally about languages and her persistance to make things succint. Dr Matthias Radestock who taught me how to write to proper programs and provided invaluable knowledge about Erlang. Thanks to Anton Stefanek for inspiring debates about programming languages, computing in general and hints for LATEX. Finally, I would like to thank my parents who always approved my, sometimes irrespon- sible and strange, decisions and allowed me to pursue my dreams. Special thanks go to my brother,Lukasz, his wife, Gosia and little Emilka for they never stopped believing in my abilities. To Natalia Contents Contents i 1 Introduction1 1.1 Contributions.................................... 3 2 Background5 2.1 Join-calculus .................................... 5 2.1.1 Historical overview............................. 6 2.1.2 Reflexive CHAM.............................. 7 2.1.3 Formal definition.............................. 8 2.2 Join-calculus implementations........................... 10 2.2.1 JoCaml ................................... 10 2.2.2 Polyphonic C# .............................. 14 2.2.3 SCHOOL and f SCHOOL......................... 19 2.2.4 Join Java .................................. 20 2.2.5 JoinHs and HaskellJoinRules ....................... 21 2.2.6 Joins Concurrency Library ........................ 24 2.2.7 Conclusion ................................. 25 2.3 Erlang........................................ 26 2.3.1 Process ................................... 27 2.3.2 Inter-process communication ....................... 28 2.3.3 Expression and Functions......................... 30 2.3.4 Exception handling............................. 30 2.3.5 Open Telecom Platform.......................... 32 2.3.6 Conclusion ................................. 34 2.4 Solving pattern-matching problems........................ 36 2.4.1 Production Rule System.......................... 36 2.4.2 RETE algorithm.............................. 36 2.4.3 Optimisations................................ 38 2.5 Data Flow Analysis................................. 39 2.5.1 Reaching Definitions Analysis....................... 39 i ii CONTENTS 2.5.2 Live Variable Analysis........................... 39 2.6 Summary ...................................... 40 3 JErlang: Formal Definition 41 3.1 Syntax........................................ 41 3.1.1 Differences between Erlang and JErlang . 44 3.2 Semantics...................................... 44 3.2.1 Operational Semantics........................... 47 3.2.2 Pattern-matching algorithms ....................... 50 3.3 Conclusion ..................................... 53 4 The language 55 4.1 Joins for Mailboxes................................. 55 4.1.1 Joins for Multiple Mailboxes ....................... 55 4.1.2 Joins for a Single Mailbox......................... 58 4.2 Language features ................................. 61 4.2.1 Getting started............................... 61 4.2.2 Joins..................................... 61 4.2.3 Order preservation in mailbox and First-Match execution . 62 4.2.4 Guards ................................... 63 4.2.5 Timeouts .................................. 64 4.2.6 Non-linear patterns............................. 65 4.2.7 Propagation................................. 66 4.2.8 Synchronous calls.............................. 67 4.2.9 OTP platform support in JErlang .................... 68 4.3 Conclusions..................................... 70 5 Implementation 71 5.1 JErlang Compiler and VM ............................ 72 5.1.1 Compiler .................................. 72 5.1.2 Built-in functions (BIFs) ......................... 73 5.1.3 Erlang's Virtual Machine ......................... 73 5.2 JErlang's VM.................................... 75 5.2.1 Search mechanism ............................. 75 5.2.2 Mailbox and Map ............................. 78 5.2.3 Locking................................... 78 5.3 Parse transform................................... 78 5.3.1 Abstract Syntax Tree ........................... 79 5.3.2 Unbounded and Bounded Variables in parse transform . 80 5.3.3 Reaching Definitions Analysis....................... 81 5.3.4 Live Variable Analysis........................... 82 5.4 Algorithms ..................................... 84 5.4.1 Standard .................................. 85 5.4.2 State space explosion............................ 89 iii 5.4.3 Lazy evaluation............................... 90 5.4.4 RETE implementation........................... 90 5.4.5 Pruning search space............................ 92 5.4.6 Patterns ordering optimisation ...................... 93 6 Evaluation 95 6.1 Correctness..................................... 95 6.1.1 Dialyzer................................... 98 6.2 Language and expressiveness ........................... 98 6.2.1 Santa Claus problem............................ 98 6.2.2 Dining philosophers problem ....................... 99 6.3 Scalability...................................... 102 6.4 Performance..................................... 104 6.4.1 Small mailboxes with quick synchronisation . 104 6.4.2 Queue size factor in joins synchronisation . 105 6.4.3 Influence of joins ordering on the performance . 107 6.4.4 Standard queues vs hash-map accelerated queues . 110 6.5 Applications..................................... 110 6.5.1 Message routing system with synchronisation on messages . 111 6.5.2 Chat system ................................112 6.6 Conclusion ..................................... 113 7 Conclusion 115 7.1 Further work .................................... 117 7.1.1 Formal definition.............................. 117 7.1.2 Performance ................................117 7.1.3 New Erlang release............................. 118 7.1.4 Benchmarks................................. 118 7.2 Closing remarks................................... 118 Bibliography 119 A Erlang Compiler Result 125 A.1 Original code.................................... 125 A.2 Abstract syntax tree for the first program.................... 126 A.3 First program expressed in Core Erlang .....................126 A.4 Assembler code (Kernel Erlang) corresponding to the first program . 128 B Parse transform result 131 B.1 Original code.................................... 131 B.2 JErlang's code resulting from running parse transform on the first program . 131 B.3 Abstract Syntax Tree representing the first program . 133 B.4 Abstract Syntax Tree resulting from running parse transform . 134 C gen joins OTP's behaviour 139 iv CONTENTS C.1 Calculator...................................... 139 C.2 Chat system..................................... 140 D Variations of the Santa Claus problem 145 D.1 Santa Claus in Erlang ............................... 145 D.2 JErlang solution to Santa Claus using popular style . 147 D.3 Sad Santa Claus problem ............................. 149 D.4 Santa Claus in gen joins ............................. 151 E Benchmarks 153 E.1 Multiple producers test-suite ...........................153 F NataMQ 155 F.1 Nata Exchange................................... 155 F.2 Nata Channel.................................... 158 F.3 Nata Publish and Subscribe............................ 160 Chapter 1 Introduction Writing concurrent applications in today's world has become not only the domain of a few programmers who produce superb, massively