UMEÅ UNIVERSITY June 4, 2021 Department of Computing Science Report

5DV199 Bachelor’s degree in computer science

Performance comparison between C and Rust compiled to WebAssembly

Magnus Medin (dv18mmn)

[email protected]

Supervisor Ola Ringdahl Contents

1 Introduction 1

1.1 Background ...... 1

1.2 Purpose and research questions ...... 2

1.3 Delimitations ...... 2

2 Related work 2

3 Theoretical background 4

3.1 Language ...... 4

3.2 Compilation ...... 5

4 Method 5

4.1 Code analysis ...... 6

4.2 Testing ...... 6

4.3 Data collection ...... 6

5 Result 7

6 Discussion 9

6.1 Limitations and further research ...... 10

6.2 Conclusion and recommendations ...... 11 Abstract

Web applications are used more and more and mainly they are developed with the help of Javascript, but when you have heavy demanding processes in a web app, javascript can be too slow. The new language WebAssembly is an alternative that seeks to solve this problem. But unlike Javascript, there are many compilers for many different languages that have WebAssembly as their compilation goal. This means that the choice of language to compile from could make a difference when it comes to execution time. This article takes a closer look at two of the most popular languages, namely C and Rust, to see if language choice makes a difference. The differences between the languages are measured with the help of tests done between a server and a client and also with the help of an analysis of the WebAssembly code. The result shows that Rust is faster than C in matrix multiplication and insertion sort but the addition test shows that the languages are similar in speed. Acknowledgement

I would like to thank William, Emil and Tobias, for inspiration, help and entertainment during these unusual study conditions. Wasm preformance comparison 1 Introduction

1 Introduction

When it comes to web applications, Javascript (JS) is top of mind for most developers. Almost all websites use it and JS has been used since the 90’s. The web has evolved since then and so has Javascript. But now when the internet is used alot, especially during a pandemic, it is becoming clearer that languages are needed that can complete and help the omnipotent programming language. Web apps biggest problem right now is that they are slow, Javascript is not made for collaboration video editing program in the cloud or a fast paced 3d shooting game in the browser. Many solutions have been presented over the years but when a solution becomes a World Wide Web Consortium recommendation and companies like Google, Mozilla, Microsoft and Apple stand behind the language it is worth taking a closer look.

WebAssembly or WASM [11] is an “Assembly” like language for the web. It defines a portable binary-code format and can be compiled from some of the most widely used programming languages, and is able to run on the most popular browsers. On the official website webassembly.org, 4 design goals are listed, which are the following.

• Efficient and fast

• Safe

• Open and debuggable

• Part of the open web platform

One thing that a lot of focus is placed on when it comes to WebAsemmly is the execution time, as this is primarily why one should choose it over JS and the creators say that WebAssembly “aims to execute at native speed”, that is, the same speed as if you were running the program locally on the computer.

1.1 Background

The compiler Emscripten [1] was announced in 2010 as an attempt for a solution on how to run native C, C++ code in the browser. This was done by compiling the code to Javascript. A subset of JavaScript called asm.js [10] that is statically-typed languages with manual memory management was announced 2013. Emscripten chose to have it as a compilation target as it had better performance than plain javascript. Here they started to

1 June 4, 2021 Wasm preformance comparison 2 Related work see the possibilities with a faster language for the web and a number of interesting programs and games were published with the help of asm.js [3] [2]. However, since JavaScript was not designed to be used as machine code there was potential for improvement if a more adapted language was used. WebAssembly was developed and released in 2017. The advantages of WASM against asm.js is that WASM can skip the parsing step in the browser’s Javscript engine. WASM has a more compact source code because it is a binary instruction format which makes fetching the code faster. Now most of the major browsers have language support for WASM, possibilities for execution outside the browser have been developed and new compilers for multiple languages have been created.

1.2 Purpose and research questions

Since the language is often used for faster execution time and that WASM can be compiled from a lot of languages, there is an interest in knowing how widely the choice of language affects the execution time. Therefore, this report focus on two of the more popular languages and make a comparison between them. Is Cs execution time faster than Rust when running on browsers using WebAssembly.

To answer this, a benchmark test for both languages is implemented, then tests are run on them, the results from the experiment are analyzed and the WASM source code is reviewed to finally present the results. A discussion about the results is made.

1.3 Delimitations

This thesis will only look at two languages. This is due to lack of time and that the results can show how and when there are differences between the languages which could be applied to further work. Another limitation is that the algorithms to be tested are of a simple nature. This is also to be within the time frame of this type of work, to make it easier to compile to Webassemble and to make the code analysis less complex.

2 Related work

WebAssembly is a new language and it should be faster than javascript. Therefore, there is a lot of new work and testing in that particular part of

2 June 4, 2021 Wasm preformance comparison 2 Related work

WASM. The previously mentioned goal of achieving native speeds has also been tested to get clarity on how close the language is to achieving this and what needs to be done to achieve the goal. The text written by Haas et al [4] describes WebAssembly and tests the speed against native C. It is a collaboration between Microsoft, Google, Apple and Mozilla where they give an overview of WebAssembly and present benchmarks where they compare WASM in comparison to native. The paper showed that WASM was close to native. But they also saw potential for improvements. WASM could take advantage of parallel compilation and utilize compilation in advance. There are javascript engines that get good results from this method. An- other thing they bring up is that one could benefit from code caching on programs that are frequently reused. But there are those who think that WASM should also be tested against larger implementations than polybench tests, Abhinav Jangda et al [8] writes “We argue that a more comprehensive evaluation of WebAssembly should rely on an established benchmark suite of large programs, such as the SPEC CPU benchmark” . In comparison with previously reported results from polybench, WASM performs slightly worse in their test with SPEC. In the report written by D. Herrera et al [6] they used the Wu-Wei version of the Ostrich benchmark suite on a number of devices and different environments. They came to the conclusion that “All browsers which incorporated WebAssembly technology showed impressive gains over their JavaScript engines.” and “WebAssembly provides excellent performance, with WebAssembly in Firefox nearing the performance of na- tive C.”. They also showed that WASM is faster than JS on all platforms and environments they tested on. The interesting part of this study is that WASM is tested on a variety of hardware such as mobiles, tablets and ARM processors and shows similar results as previous studies.

There are many articles that test Rust in comparison to C, in the article [7] Jabber examins some popular sorting algorithms against each other and the author presents a result where Rust is faster in all tests except when sorting with insertion sort. In the conclusion, the author writes “It’s quite difficult to say which language is faster because it depends on case to case. But we can say that Rust is a competitor of C in terms of speed and it is faster than many other popular languages like Java and Python.“

There are many related work that look at WebAssembly in comparison to Javascript or native code on many different platforms. There are also many articles that benchmark the languages against each other but when it comes to comparisons between implementations within WebAssembly, there is not much done. An article addressing the topic is published by IBM written by Josh Hannaford [5] where a team looked at which language choice was right for their work with WASM and Node.js. The conclusion was that Rust was the best choice for them “Another major advantage of using Rust with Wasm is the execution speed and the binary size. Both of these are comparable or

3 June 4, 2021 Wasm preformance comparison 3 Theoretical background even a little better than other lower-level, non-garbage collected languages like C and C++.”.

3 Theoretical background

WebAssembly is intended as a language similar to assembly for the Web, however, it is starting to move away from web only and is now possible to run as a regular program locally on the computer. The language is not exactly an assembly language but rather similar to machine code that can be read by the browser. The code is often presented in an assembly format so called “WebAssembly text format”. The code executes using a virtual machine that runs in the browser, which makes the computer’s hardware less important for compatibility. When you compile from a language to WebAssembly, the compiler creates a binary .wasm file, a so-called WebAssembly module. The Wasm file is then the file that is downloaded from a website to run as a program in the browser. The file is then fetched from the web page’s front end using javascript and a web assembly module is returned which can then be used to call on the functions within the module.

3.1 Language

The language is very low level and therefore the syntax is very comprehen- sive. The previously mentioned module holds functions, globals, tables, and memory. Memory is shared memory that both WASM and JS can see. Glob- als behave like memory but are only one value. A module can export parts of itself to make the functions public.

(export "add" (func $ add))

This add function can be used by JS when the module is fetched for use by, for example, a website. The module can also import parts from Javascript. The language contains of 10 different types of instructions such as numeric in- structions for simple mathematics and logical operators, control instructions for flow of control. In addition, there are instructions for memory, tables and values as well as handling null.

4 June 4, 2021 Wasm preformance comparison 4 Method

1 (module

2

3 (export "add" (func $add))

4

5 (func $add (param $x i32) (param $y i32) (result i32) 6 local.get $x 7 local.get $y 8 i32.add 9 ) 10 )

The code is an example of a module with an add function. The language is stack based so the code is for example, retrieve the value A = 5 retrieve the value B = 4. Add uses two values so it pops the stack with the values and pushes the answer 9 back to the stack.

3.2 Compilation

For compilation to WASM, both Rust and Emscript use LLVM [9]. The process of translating Rust and C into machine code takes place in 8 steps: Sorce, Scanner, Parsing, Intermediate Representation, IR Optimizer, Seman- tic Analysis, Target Code and Compiled code. Emscripten and Rust then use LLVM’s APIs to optimize the process specifically for their purpose.

4 Method

The method can be divided into two separate parts. The first part is a pure code analysis of the results from the different languages compiled to WASM to get a picture of how the languages differ in writing. The second part is a test testing execution for the same code to get a picture of how the languages differ in practical terms and in terms of perceived time. The testing is inspired by Haas et al [4] rather than other related work because it fits well with the delimitations and that it is a well-regarded text. The implemented tests therefore strive to resemble existing tests in polybenchC whenever possible.

The implementations in both languages have tried to follow what is consider to be standard for the language. By looking at different code and different guides for different parts.

An example of this is in the initialization of arrays in Rust. The most common way is to initialize arrays with the value 0 at each position, however, there are

5 June 4, 2021 Wasm preformance comparison 4 Method ways to let the positions hold garbage data using the uninitialized function in Rust. The test code for this thesis chose the slower way that is considered to be normal instead of what is considered unsafe and faster.

4.1 Code analysis

The total number of lines of code are counted for the WASM modules. Then the same thing is done for the function that is tested. The function is then analyzed in more depth by analyzing the occurrence of various instructions in the code. The result is then compared between the languages.

4.2 Testing

A test program that measures the time for fetching the WASM module as well as the time of execution of the test function is implemented in JS and run on a server. The tests then consisted of a client contacting the server and running the test 40 times for each language and implementation. The values from the tests are then used to produce an average value for fetching time as well as execution time for each WASM file. Each implementation is tested in the same session for the different languages to minimize external factors such as internet speed and traffic on the servers. Between individual tests, the browser’s cache is also cleaned so that no information from previous tests is saved. The test run with three different implementations for each language.

• Matrix multiplication

– which took two 100X100 arrays with values 1-10,000 and multi- plied them together.

• Insertion sort

– with int values between 0-999 in an array of size 1000.

• Add

– A function that took two values and added them together.

4.3 Data collection

The test was implemented in JS and was run using Apache httpd 2.4.46 on Amazon web service Ec2 located in Stockholm. The client ran the tests in the Firefox browser. The specifications for the server and the client are

6 June 4, 2021 Wasm preformance comparison 5 Result

Server OS: Ubuntu Server 20.04 LTS Processors: t3.micro (- ECUs, 2.5 GHz, 1 GiB memory, EBS only) Nr of processors: 2 vCPU Network: vpc-0ed75a67

Client OS: Manjaro Linux 21.0.4 Processors: AMD Ryzen 7 4700U 2.0 GHz Nr of processors: 1 Network: Intel Corporation Wi-Fi 6 AX200

5 Result

The results from the code analysis can be seen in Table 1 and 2. Where mm is matrix multiplication, sort is insertion sort and add is the add function.

Table 1: Number of lines of code for each test Code analysis Algorithm Number of lines Number of lines in module in function Rust MM 289 244 C MM 863 672 Rust Sort 113 86 C Sort 580 367 Rust ADD 8 7 C ADD 219 57

Table 2: Number of instructions for each test Code analys Algorithm get set add store load offset memory tee Rust MM 59 34 40 3 2 0 0 0 C MM 270 176 23 31 43 69 0 0 Rust Sort 21 11 12 3 2 0 0 8 C Sort 140 105 12 13 22 23 0 0 Rust ADD 2 0 3 0 0 0 3 0 C ADD 12 6 3 0 0 4 0 0

7 June 4, 2021 Wasm preformance comparison 5 Result

Figure 1-3 shows a comparison between the two languages for each test in ms. The red bar is fetch time while the blue bar is execution time. The p-value is calculated with a two-sample t-test with a significance level of 5%.

Figure 1: Matrix multiplication with a p-value of p = 2.5836 × 10−9

Figure 2: Insertion sort with a p-value of p = 4.6593 × 10−5

8 June 4, 2021 Wasm preformance comparison 6 Discussion

Figure 3: ADD with a p-value of p = 0.7315

Table 3: Results from all tests Algorithm Min Mean Max C MM 133 255.5500 650 Rust MM 189 134.9250 189 C Sort 92 143.2381 212 Rust Sort 60 109.5122 197 C Add 44 78.9500 141 Rust Add 49 80.6000 148

6 Discussion

From the tests, The biggest difference between the languages is in matrix multiplication. While the result does not look similar to Jabber [7] as Rust in the test with Insertion sort actually takes less time. The result from the test Add is very even between the languages. The result from Josh Hannaford’s article [5] is not so clearly presented but the results in this thesis still show similar results as he describes. The code analysis shows that Rust in all tests has the shorter code with fewer instructions both when comparing modules and when comparing the functions. Furthermore, the code analysis show that the ratio of instructions used differs between the languages, this means that there is actually a difference between them. A correlation that is visible is that C makes more calls to the shared memory between JS and WASM for

9 June 4, 2021 Wasm preformance comparison 6 Discussion matrix multiplication and insertion sort, while for add, it is Rust that does the most. The fetching for the modules looked very similar between all tests and languages. The tests together with the code analysis show an expected result. When a language uses more instructions, the execution takes longer, especially when there are instructions for managing the shared memory. The result does not show why the compilers choose to translate native to WASM with the instructions they do, it only shows how they chose to translate the code. The overhead for C, it is much larger and C needs more help code to run the programs in WASM. This may be a product of the fact that when C was developed, WASM was not intended at all and that the Emscripten compiler is not specific to C while the Rust compiler is. Another thing that may be a reason for the longer code is that Emscripten recently changed backend to LLVM and that the optimization may not be fully adapted yet or that Rust translates the code using LLVM’s API in a more optimal way. Why the Add test is slower for Rust can not be due to the overhead as C also here has more code in the module and function. But the Rust compiler chooses to use more instructions for the shared memory than the Emscripten which may be a reason. Another reason may be that the modules in this test are significantly smaller, which means that there is less difference between them in pure length. Even though the P-value here is high, it is lower than in the other tests, which can also mean that chance has a greater impact.

6.1 Limitations and further research

The tests are on individual algorithms with a very specific structure. If one were to follow the recommendation from Abhinav Jangda et al [8], broader and larger testing would be required to get more accurate results. But a first step is to test very simple algorithms to detect differences between languages. Even on the simpler algorithms there is more testing one could do, an in- teresting job would have been to try to translate the testing as described by Haas et al [4] from C to Rust and thus get results from a more standardized test. Another thing that would have made the result more secure would have been to follow the testing in D. Herrera et al [6] paper and change hardware and software on both the server and the client to see if it makes any differ- ence. To get a better answer to exactly why there is a difference, a deeper analysis of the compilers can be made and examine why the choices made are actually made and further examine the browser engine to see if there is any difference between the instructions in the execution.

10 June 4, 2021 Wasm preformance comparison 6 Discussion

6.2 Conclusion and recommendations

According to the results from the report, there is a difference between C and Rust when compiled for WebAssembly. Since there is a difference between the languages, there is also the possibility that there is a better choice when you want as short of an execution time as possible. The result shows that Rust is preferable for more demanding calculations such as matrix multiplication and insertion sort. For shorter and simpler tasks such as addition, it is not as clear which language to choose. The result also shows that it is in the browser engine that the difference occurs rather than transferring the files from the server. The result does not tell which criteria should be met for choosing a particular language but more calls to the shared memory correlate with slower execution. The result don’t show that one language is always preferable to another, but rather that there is a variation and difference between the languages.

11 June 4, 2021 Wasm preformance comparison References

References

[1] Emscripten Community. Emscripten. https://emscripten.org/. Ac- cessed: 2021-03-30. [2] DOSBox crew. DOSBox ported to Emscripten. https://github.com/ dreamlayers/em-dosbox. Accessed: 2021-03-30. [3] gjimenez. ‘Epic Citadel’ Demo Shows the Power of the Web as a Plat- form for Gaming. https://blog.mozilla.org/futurereleases/ 2013/05/02/epic-citadel-demo-shows-the-power-of-the-web- as-a-platform-for-gaming/. Accessed: 2021-03-30. [4] Andreas Haas et al. “Bringing the Web up to Speed with WebAssem- bly”. In: Proceedings of the 38th ACM SIGPLAN Conference on Pro- gramming Language Design and Implementation. PLDI 2017. Barcelona, Spain: Association for Computing Machinery, 2017, pp. 185–200. isbn: 9781450349888. doi: 10.1145/3062341.3062363. [5] Josh Hannaford. Why using WebAssembly and Rust together improves Node.js performance. https://developer.ibm.com/technologies/ web-development/articles/why-webassembly-and-rust-together- improve-nodejs-performance/#. Accessed: 2021-06-01. [6] D. Herrera, H. Chen, and Erick Lavoie. “WebAssembly and JavaScript Challenge: Numerical program performance using modern browser tech- nologies and devices”. In: 2018. [7] Jabber. Speed of Rust vs C. https://kornel.ski/rust- c- speed. Accessed: 2021-05-19. [8] Abhinav Jangda et al. “Not So Fast: Analyzing the Performance of WebAssembly vs. Native Code”. In: 2019 USENIX Annual Technical Conference (USENIX ATC 19). Renton, WA: USENIX Association, July 2019, pp. 107–120. isbn: 978-1-939133-03-8. [9] C. Lattner and V. Adve. “LLVM: a compilation framework for life- long program analysis transformation”. In: International Symposium on Code Generation and Optimization, 2004. CGO 2004. 2004, pp. 75–86. doi: 10.1109/CGO.2004.1281665. [10] Unknown. asm.js. http://asmjs.org/. Accessed: 2021-03-30. [11] WebAssembly. webassembly.org. https : / / webassembly . org/. Ac- cessed: 2021-03-30.

12 June 4, 2021