Evaluating Research Tool Usability on the Web
Total Page:16
File Type:pdf, Size:1020Kb
Research.js: Evaluating Research Tool Usability on the Web Joel Galenson, Cindy Rubio-Gonzalez,´ Sarah Chasins, Liang Gong University of California, Berkeley fjoel,rubio,schasins,[email protected] Abstract To make research tools available online, current practice requires researchers to set up and run their own servers, Many research projects are publicly available but rarely used which run their tools on users’ inputs. Setting up such a due to the difficulty of building and installing them. We server can be a difficult, time-consuming, and costly pro- propose that researchers compile their projects to JavaScript cess. Because of load restrictions, this approach imposes a and put them online to make them more accessible to new limit on the number of concurrent users. Further, most re- users and thus facilitate large-scale online usability studies. search tools are not designed to be secure, which makes it 1. Motivation dangerous to run them on a server that accepts user inputs. When researchers create new programming languages and Some researchers also distribute their tools as virtual ma- tools, they sometimes choose to release them publicly. Un- chine images, but these are very heavyweight. fortunately, released projects are often complicated to in- We propose helping researchers compile their tools to stall, which makes it difficult to find enough users to con- JavaScript, thereby allowing anyone with a web browser to duct large-scale usability studies. Yet releasing tools that are use them. This would enable researchers to make their tools accessible to users is typically difficult for researchers. We available online without running their own secure servers. propose building an infrastructure that makes public releases The relatively-lightweight client-side JavaScript would com- easy for both potential users and researchers, thereby facili- plete all processing, eliminating researchers’ security con- tating large-scale user studies. cerns. Further, the tools can be distributed within static web- Unfortunately, even when languages and tools are pub- pages, which can easily be hosted on any number of exter- licly available, they are often difficult to build and install. nal services, so researchers need not run servers themselves. A recent conference encouraged authors to submit artifacts With this approach, users could browse language and tool and stipulated that the setup should take less than 30 min- demos as easily as they browse web pages, trying tools be- utes [6]. For some potential users, including reviewers and fore deciding to install them locally. If users consent, their researchers who intend to use or evaluate the artifacts, this usage data could be sent to the researchers, allowing each may be a reasonable amount of time. However, this level of visitor to serve as a participant in a user study. The collected time commitment is likely to prevent a potential user with a data could then be leveraged to evaluate and ultimately im- passing interest from trying a programming language or tool. prove usability. In essence, the online release becomes a In order to keep these users interested and collect data about large-scale remote user study. their usage, we must lower the barriers to entry. Making it The research community has already started advocating easy for these moderately interested parties to participate is releases. Conferences increasingly encourage authors to sub- key to achieving truly large-scale user studies. mit artifacts [1] to allow others to evaluate and build on their In recent years, the web has become an increasingly pow- contributions. While this too is a laudable goal, we focus not erful platform, one on which applications are easily portable. on reproducing evaluations but on the potential of public re- Some research projects have made online versions of their leases to bring large-scale remote user studies within reach. tools available, making them easy to explore (e.g., [3, 7]). 2. Proposed Approach Permission to make digital or hard copies of all or part of this work for personal or We propose building an infrastructure that makes it easy to classroom use is granted without fee provided that copies are not made or distributed compile existing projects to JavaScript and optionally collect for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the usage data. We plan to leverage existing tools to translate author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or programs into JavaScript. One such tool is Emscripten [15], republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. which compiles C/C++ code and LLVM [14] bitcode to PLATEAU ’14, October 21, 2014, Portland, OR, USA.. JavaScript. There are other projects that compile many lan- Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-2277-5/14/10. $15.00. guages, including Scala, Haskell, and Python, directly to http://dx.doi.org/10.1145/2688204.2688217 JavaScript [2, 5]. 53 Changes required com/jgalenson/research.js, which links to demos for MiniSat, Boolector, and Hugs. Name Source Build Compiles Runs We analyzed the performance of our JavaScript versions MiniSat [13] 0 (0) 1 (1) 4 4 of MiniSat, Lingeling, and Boolector on a few benchmarks Lingeling [10] 1 (1) 0 (0) 4 4 hand-chosen from SATLIB and SMT-LIB 2. On average, Boolector [11] 0 (0) 0 (0) 4 4 the Emscripten-compiled versions are 3 times slower than Hugs [4] 1 (1) 1 (1) 4 4 native, which we believe is sufficient for our purposes. To Z3 [12] 1 (1) 1 (1) 4 8 show that performance is likely to improve further over time, LLVM+Clang [14] 12 (5) 3 (3) 4 8 we compiled and tested MiniSat with eighteen-month-old Table 1. The projects we have attempted to compile with versions of Emscripten and Firefox and found that it was 11 Emscripten, the number of changes required to compile them times slower than native. Using our prototype, compile times (the number of lines of code and files changed), and whether are 2-7 times slower and file sizes of Emscripten-compiled they successfully compile and run. projects are less than 2 times larger. Acknowledgments An alternative plan for running research tools in a web browser is to use interpreters written in JavaScript [9] with- We would like to thank Christos Stergiou, Nishant Totla, and out compiling the project under evaluation. As a last resort, Cuong Nguyen for their useful advice. We also appreciate projects could be run in a JavaScript PC emulator [8]. the valuable comments from anonymous reviewers and our In our infrastructure, we also plan to collect and analyze shepherd Joshua Sunshine for improving the paper. traces automatically (with users’ consent) to help evaluate References projects’ usability and guide future improvements. [1] Artifact Evaluation for Software Conferences. http://www. There are several issues to consider when translating re- artifact-eval.org/. Accessed: 09/10/2014. search projects into JavaScript and running in a browser. [2] Emscripten. https://github.com/kripken/ Closed-Source Binaries. Many projects use closed-source emscripten/wiki. Accessed: 11/22/2013. binaries. It may be possible to compile assembly code to [3] Flapjax. http://www.flapjax-lang.org/. Accessed: JavaScript or decompile binaries into a higher-level language 07/03/2013. that could be compiled directly. Alternatively, we could pro- [4] Hugs. http://www.haskell.org/hugs/. Accessed: vide an option to obfuscate the generated JavaScript code. 11/22/2013. Libraries. A research project might depend on many exter- [5] List of languages that compile to JS. https: nal libraries, each of which would have to be compiled with //github.com/jashkenas/coffee-script/wiki/ the above techniques. We believe that pre-compiling and es- List-of-languages-that-compile-to-JS. Accessed: tablishing a central repository that hosts all known ported 11/22/2013. libraries would help alleviate this problem. [6] OOPSLA Artifacts. http://splashcon.org/2013/cfp/ 665. Accessed: 07/03/2013. Performance. Running a research project in a browser is slower than running it natively. Fortunately, performance [7] rise4fun. http://rise4fun.com/. Accessed: 07/03/2013. is not generally critical for evaluating usability. Thus we [8] JavaScript PC Emulator. http://bellard.org/jslinux/. prioritize compatibility and ease-of-use over performance. Accessed: 07/03/2013. [9] repl.it. http://repl.it/. Accessed: 07/03/2013. 3. Exploration and Early Experience [10] A. Biere. Lingeling, plingeling, picosat and precosat at sat In our prototype, we use Emscripten [15], which is designed race 2010. FMV Report Series Technical Report, 10(1), 2010. to allow execution at close to native speeds. Many projects [11] R. Brummayer and A. Biere. Boolector: An efficient smt have been ported to JavaScript with Emscripten, including solver for bit-vectors and arrays. In TACAS, pages 174–177. Unreal Engine 3, LaTeX, Lua, Python, and parts of LLVM Springer, 2009. and Emscripten [2, 5]. It is robust and, with its emphasis [12] L. De Moura and N. Bjørner. Z3: An efficient smt solver. In on porting games to the web, performs well. We have used TACAS, pages 337–340. Springer, 2008. it to compile the tools, such as compilers and SAT/SMT [13] N. Een´ and N. Sorensson.¨ An extensible sat-solver. In solvers, listed in Table 1. Only one project required changing Theory and Applications of Satisfiability Testing, pages 502– 1 more than a single line of code. Our prototype does not yet 518. Springer, 2004. automatically generate code to collect tool usage statistics. [14] C. Lattner and V.