c J Functional Programming 1 January Cambridge University Press Benchmarking Implementations of Functional Languages with Pseudoknot a FloatIntensive Benchmark Pieter H Hartel Marc Feeley Martin Alt Lennart Augustsson Peter Baumann Marcel Beemster Emmanuel Chailloux Christine H Flo o d Wolfgang Grieskamp John H G van Groningen Kevin Hammond Bogumil Hausman Melo dy Y Ivory Richard E Jones Jasp er Kamp erman Peter Lee Xavier Leroy Rafael D Lins Sandra Lo osemore Niklas Rojemo Manuel Serrano JeanPierre Talpin Jon Thackray Stephen Thomas Pum Walters Pierre Weis Peter Wentworth Abstract Over implementation s of dierent functional languages are b enchmarked using the same program a oating p oint intensive application taken from molecular biology The principal asp ects studied are compile time and Dept of Computer Systems Univ of Amsterdam Kruislaan SJ Amsterdam The Netherlands email pieterfwiuvanl Depart dinformatique et ro Univ de Montreal succursale centreville Montreal HC J Canada email feeleyiroumontrealca Informatik Universitat des Saarlandes Saarbruc ken Germany email altcsunisbde Dept of Computer Systems Chalmers Univ of Technology Goteb org Sweden email augustsscschalmersse Dept of Computer Science Univ of Zurich Winterthurerstr Zurich Switzerland email baumanniunizh ch Dept of Computer Systems Univ of Amsterdam Kruislaan SJ Amsterdam The Netherlands email b eemsterfwiuvanl LIENS URA du CNRS Ecole Normale Superieure rue dUlm PARIS Cedex France email EmmanuelChaillou xensfr Lab oratory for Computer Science MIT Technology Square Cambridge Massachusetts USA email chflcsmitedu Berlin University of Technology Franklinstr Berlin Germany email wgcstub erlinde Faculty of Mathematics and Computer Science Univ of Nijmegen To erno oiveld ED Nijmegen The Netherlands email johnvgcskunnl Dept of Computing Science Glasgow University Lilybank Gardens Glasgow G QQ UK email khdcsglasgowacuk Computer Science Lab Ellemtel Telecom Systems Labs Box S Alvsjo Sweden email b ogdanerixericssonse Computer Research Group Institute for Scientic Computer Research Lawrence Livermore National Lab ora tory P O Box L Livermore CA email ivoryllnl gov Dept of Computer Science Univ of Kent at Canterbury Canterbury Kent CT NF UK email REJonesukcacuk CWI Kruislaan SJ Amsterdam The Netherlands email jasp ercwinl Dept of Computer Science Carnegie Mellon University Forb es Avenue Pittsburgh Pennsylvania USA email p etelcscmuedu INRIA Ro cquencourt pro jet Cristal BP Le Chesnay France email XavierLeroyinriafr Departamento de Informatica Universidade Federal de Pernambuco Recife PE Brazil email rdldiufp ebr Dept of Computer Science Yale Univ New haven Connecticut email lo osemoresandracsyal eedu Dept of Computer Systems Chalmers Univ of Technology Goteb org Sweden email ro jemocschalmersse INRIA Ro cquencourt pro jet Icsla BP Le Chesnay France email ManuelSerranoin ria fr Europ ean ComputerIndustry Research Centre Arab ella Strae D Munich Germany email jpecrcde Harlequin Ltd Barrington Hall Barrington Cambridge CB RG England email jontharlequincouk Dept of Computer Science Univ of Nottingham Nottingham NG RD UK email sptcsnottacuk CWI Kruislaan SJ Amsterdam The Netherlands email pumcwinl INRIA Ro cquencourt pro jet Cristal BP Le Chesnay France email PierreWeisinriafr Dept of Computer Science Rho des Univ Grahamstown South Africa email cspwcsruacza Hartel Feeley et al execution time for the various implementation s that were b enchmarked An imp ortant consideration is how the program can b e mo died and tuned to obtain maximal p erformance on each language implementation With few exceptions the compilers take a signicant amount of time to compile this program though most compilers were faster than the then current GNU C compiler GCC version Compilers that generate C or Lisp are often slower than those that generate native co de directly the cost of compiling the intermediate form is normally a large fraction of the total compilation time There is no clear distinction b etween the runtime p erformance of eager and lazy implementations when appro priate annotations are used lazy implementations have clearly come of age when it comes to implementing largely strict application s such as the Pseudoknot program The sp eed of C can b e approached by some implementa tions but to achieve this p erformance sp ecial measures such as strictness annotations are required by nonstrict implementations The b enchmark results have to b e interpreted with care Firstly a b enchmark based on a single program cannot cover a wide sp ectrum of typical application s Secondly the compilers vary in the kind and level of optimisation s oered so the eort required to obtain an optimal version of the program is similarly varied Intro duction At the Dagstuhl Workshop on Applications of Functional Programming in the Real World in May Giegerich and Hughes several interesting applications of functional languages were pre sented One of these applications the Pseudoknot problem Feeley et al had b een written in several languages including C Scheme Rees and Clinger Multilisp Halstead Jr and Miranday Turner A numb er of workshop participants decided to test their compiler technology using this particular program The rst p oint of comparison is the sp eed of compilation and the sp eed of the compiled program The second p oint is how the program can b e mo died and tuned to obtain maximal p erformance on each language implementation available The initial b enchmarking eorts revealed imp ortant dierences b etween the various compilers The rst impression was that compilation sp eed should generally b e improved After the workshop we have continued to work on improving b oth the compilation and execution sp eed of the Pseudoknot program Some researchers not present at Dagstuhl joined the team and we present the results as a record of a small scale but exciting collab oration with contributions from many parts of the world As is the case with any b enchmarking work our results should b e taken with a grain of salt We are using a realistic program that p erforms a useful computation however it stresses particular language features that are probably not representative of the applications for which the language implementations were intended Implementations invariably tradeo the p erformance of some programming features for others in the quest for the right blend of usability exibility and p erformance on typical applications It is clear that a single b enchmark is not a go o d way to measure the overall quality of an implementation Moreover the p erformance of an implementation usually but not always improves with new releases as the implementors repair bugs add new features and mo dify the compiler We feel that our choice of b enchmark can b e justied by the fact that it is a real world application that it had already b een translated into C and several functional languages and that we wanted to compare a wide range of languages and implementations The main results agree with those found in earlier studies Cann Hartel and Langendo en Section briey characterises the functional languages that have b een used The compilers and inter preters for the functional languages are presented in Section The Pseudoknot application is intro duced in Section Section describ es the translations of the program to the dierent programming languages The b enchmark results are presented in Section The conclusions are given in the last section y Miranda is a trademark of Research Software Ltd Pseudoknot benchmark language source ref typing evaluation order match SML family Caml INRIA Weis strong p oly eager higher pattern SML Committee Milner et al strong p oly eager higher pattern Haskell family Clean Nijmegen Plasmeijer and strong p oly lazy higher pattern van Eekelen Gofer Yale Jones strong p oly lazy higher pattern LML Chalmers Augustsson and strong p oly lazy higher pattern Johnsson Miranda Kent Turner strong p oly lazy higher pattern Haskell Committee Hudak et al strong p oly lazy higher pattern RUFL Rho des Wentworth strong p oly lazy higher pattern Lisp family Common Committee Steele Jr dynamic eager higher access Lisp Scheme Committee Rees and Clinger dynamic eager higher access Parallel and concurrent languages Erlang Ericsson Armstrong et al dynamic eager rst pattern Facile ECRC Thomsen et al strong p oly eager higher pattern ID MIT Nikhil strong p oly eager higher pattern nonstrict Sisal LLNL McGraw et al strong mono eager rst none Intermediate languages CMC Recife Lins and Lira strong p oly lazy higher access Stoel Amsterdam Beemster strong p oly lazy higher case Other functional languages Epic CWI Walters and strong p oly eager rst pattern Kamp erman Opal TU Berlin Didrich et al strong p oly eager higher pattern Trafola Saarbruc ken Alt et al strong p oly eager higher pattern C ANSI C Committee Kernighan and weak eager rst none Ritchie Table Language characteristics The source of each language is fol lowed by a key reference to the language denition The remaining columns characterise the typing discipline the evaluation strategy whether the language is rst or higherorder and the patternmatching facilities Hartel Feeley et al Languages The Pseudoknot b enchmark takes into account a large numb er of languages and an even larger numb er of compilers Our aim has b een to cover as comprehensively as p ossible the landscap e of functional languages while emphasising typ ed languages