Bachelor Degree Project Webassembly Vs. Its Predecessors

Home , ANGLE (software), Browser security, Emscripten, HTML5, List of Mozilla products, List of web browsers

Bachelor Degree Project

WebAssembly vs. its predecessors A comparison of technologies

Author: Stefan Fredriksson Supervisor: Jesper Andersson Semester: VT 2020 Subject: Computer Science Abstract For many years it has only been HTML, CSS, and JavaScript that have been native to the Web. In December 2019, WebAssembly joined them as the fourth language to run natively on the Web. This thesis compared WebAssembly to the technologies ActiveX, Java applets, Asm.js, and Portable Native Client (PNaCl) in terms of their performance, security, and browser support. The reason why this was an interesting topic to investigate was to determine in what areas WebAssembly is an improvement over previous similar technologies. Another goal was to provide companies that still use older technologies with an indication as to whether or not it is worth upgrading their system with newer technology. To answer the problem, the thesis mainly focused on comparing the performance of the technologies through a controlled experiment. The thesis also aimed at getting a glimpse of the security differences between the technologies by the use of a literature study. The thesis showed that PNaCl was the technology with the best performance. However, WebAssembly had better browser support. Also, PNaCl is deprecated while WebAssembly is heavily supported and could potentially be further optimized.

Keywords: WebAssembly, wasm, ActiveX, Java applet, applet, Asm.js, Portable Native Client, PNaCl, Performance, Security, Browser support, Dynamic Web Preface

I would like to thank my supervisor during this thesis, Jesper Andersson, for guiding me and coming up with ideas I would not have had without him. I would also like to thank my peers who reviewed the thesis during its different stages. Contents

List of Figures

List of Tables

List of Listings

1 Introduction1 1.1 Background...... 1 1.2 Related work...... 2 1.3 Problem formulation...... 3 1.4 Objectives...... 4 1.5 Scope/Limitation...... 4 1.6 Target group...... 4 1.7 Outline...... 4

2 Method5 2.1 Scientiﬁc Approach...... 5 2.1.1 Literature study...... 5 2.1.2 Controlled Experiment...... 5 2.2 Reliability and Validity...... 6

3 Dynamic web performance7 3.1 Dynamic web...... 7 3.2 Technologies server-side...... 8 3.3 Technology overview...... 8 3.3.1 Java Applets...... 8 3.3.2 ActiveX...... 9 3.3.3 Asm.js...... 10 3.3.4 Portable Native Client...... 11 3.3.5 WebAssembly...... 12 3.4 Performance Testing...... 12

4 Data Collection 14 4.1 Design...... 14 4.1.1 Design for collecting performance data...... 14 4.1.2 Design for collecting data on security...... 17 4.2 Performance experiment preparations...... 18 4.2.1 Browser settings...... 18 4.2.2 Compilation tools...... 18 4.3 Performance test execution...... 19 4.3.1 Readying machine for testing...... 19 4.3.2 Running the tests...... 20 4.3.3 Hardware and Software...... 20

5 Results 22 5.1 Performance experiment...... 22 5.1.1 Result explanation...... 22 5.1.2 Fibonacci application...... 22 5.1.3 Array application...... 25 5.1.4 Numeric application...... 28 5.2 Qualitative results...... 31 5.2.1 Security...... 31 5.2.2 Browser support...... 34

6 Analysis 35 6.1 Performance experiment...... 35 6.1.1 Execution time...... 35 6.1.2 Load time...... 38 6.1.3 CPU usage...... 41 6.1.4 Memory usage...... 44 6.2 Qualitative results...... 47 6.2.1 Security...... 47 6.2.2 Browser support...... 48

7 Discussion 49 7.1 Execution time & Load time...... 49 7.2 CPU & RAM Usage...... 50 7.3 Security...... 50 7.4 Browser support...... 51 7.5 Summary...... 51

8 Conclusions & Future work 53

References 55

A Appendix 1A

B Appendix 2B

C Appendix 3C List of Figures

3.1 High-level flow of a server-side scripting application...... 7 3.2 High-level flow of a client-side scripting application...... 8 3.3 High-level flow of a Java applet application...... 9 3.4 High-level flow of an ActiveX application...... 10 3.5 High-level flow of an Asm.js application...... 11 3.6 High-level flow of a Portable Native Client application...... 11 3.7 High-level flow of a WebAssembly application...... 12 4.1 Design of the test process flow...... 17 5.1 Execution times of the Fibonacci application...... 23 5.2 Load times of the Fibonacci application...... 23 5.3 CPU usage of the Fibonacci application running on the desktop...... 24 5.4 CPU usage of the Fibonacci application running on the laptop...... 24 5.5 Memory usage of the Fibonacci application running on the desktop.... 25 5.6 Memory usage of the Fibonacci application running on the laptop..... 25 5.7 Execution times of the Array application...... 26 5.8 Load times of the Array application...... 26 5.9 CPU usage of the Array application running on the desktop...... 27 5.10 CPU usage of the Array application running on the laptop...... 27 5.11 Memory usage of the Array application running on the desktop...... 28 5.12 Memory usage of the Array application running on the laptop...... 28 5.13 Execution times of the Numeric application...... 29 5.14 Load times of the Numeric application...... 29 5.15 CPU usage of the Numeric application running on the desktop...... 30 5.16 CPU usage of the Numeric application running on the laptop...... 30 5.17 Memory usage of the Numeric application running on the desktop..... 31 5.18 Memory usage of the Numeric application running on the laptop...... 31 6.1 Box plot showing the variance of execution times for the different technologies running the Fibonacci application...... 36 6.2 Box plot showing the variance of execution times for the different technologies running the array application...... 37 6.3 Box plot showing the variance of execution times for the different technologies running the numeric application...... 38 6.4 Box plot showing the variance of load times for the different technologies running the Fibonacci application...... 39 6.5 Box plot showing the variance of load times for the different technologies running the array application...... 40 6.6 Box plot showing the variance of load times for the different technologies running the numeric application...... 41 6.7 Box plot showing the variance of CPU usage for the different technologies running the Fibonacci application...... 42 6.8 Box plot showing the variance of CPU usage for the different technologies running the array application...... 43 6.9 Box plot showing the variance of CPU usage for the different technologies running the numeric application...... 44 6.10 Box plot showing the variance of memory usage for the different technologies running the Fibonacci application...... 45 6.11 Box plot showing the variance of memory usage for the different technologies running the array application...... 46 6.12 Box plot showing the variance of memory usage for the different technologies running the numeric application...... 47 C.1 Execution times of the Fibonacci application showing the execution time per browser...... C C.2 Load times of the Fibonacci application showing the load time per browser.C C.3 Execution times of the Array application showing the execution time per browser...... D C.4 Load times of the Array application showing the load time per browser..D C.5 Execution times of the Numeric application showing the execution time per browser...... E C.6 Load times of the Numeric application showing the load time per browser.E List of Tables

4.1 Keywords used during the literature study...... 18 4.2 Speciﬁcation of the machines that the performance tests were executed on. 20 4.3 The versions of the software used in this study...... 21 5.1 Browser support for each technology...... 34 Listings

1 Installing the Emscripten SDK...... A 2 WebAssembly compilation command for the array application...... A 3 Makefile of the array PNaCl application...... A 4 Installing the NaCl SDK...... B 5 Make.bat file content of a PNaCl application...... B 6 PNaCl compilation commands...... B 7 Installing curl for Visual Studio 2017...... B This thesis is the final product of a bachelor degree in computer science consisting of 15 credits at the Linnaeus University.

1 Introduction

The Web is the largest and still growing platform in most aspects of today’s industries, and 95% of web sites use JavaScript as their front-end language[1]. The web sites of today are more demanding than the ones of the past with the web sites of today becoming closer to native applications. Today’s web sites can include things such as 3D visualization, audio- and video editing, and 3D video games. However, JavaScript was not meant to handle such applications, and thus developers have tried many times to develop technologies for the Web that can handle such applications. Recently, one of these technologies, WebAssembly, was declared a web standard as it runs natively in web browsers. This thesis researched what WebAssembly does differently to some of the earlier attempts at implementing native-speed applications on the Web. The technologies that were compared to WebAssembly were Java applets, ActiveX, asm.js, and Portable Native Client.

1.1 Background Back before JavaScript existed, the Web was a much more static place, where web pages did not do much other than display information. In 1995 when the Java Programming Language was released and with it the Java Applets, the possibility of embedding small applications on a web page became a reality. These small applications made it possible to run visualizations and games on a web page[2]. The Java applets were a massive success, and others wanted to share that success. In the same year as the release of the Java applets, Brendan Eich created the JavaScript language in ten days[3]. In 1996 Microsoft joined in by releasing ActiveX for their Inter- net Explorer web-browser. Like the Java applets, ActiveX allowed developers to embed small applications on a web page that could perform the same functionality as native applications[4]. Java applets ruled the web industry for over a decade, only challenged by Microsoft, but were eventually replaced with JavaScript as the web-browsers started using compilers aimed at JavaScript[5]. While JavaScript has improved in performance over the years, it still comes up short when put under heavy pressure. Mozilla tried to improve the performance of JavaScript in 2013 by making it a compile-target through asm.js. By making it a compile-target, it allowed developers to write their applications in lower-level languages such as C and C++ and then compile the code into optimized JavaScript code through a tool such as Emscripten[6]. Google attempted to increase the performance of applications running on web pages through the use of their Native Client (NaCl) and later the Portable Native Client (PNaCl)[7]. In 2015 the World Wide Web Consortium (W3C) announced they were working on WebAssembly with help from all major browser vendors. It was later released in 2017, and in December 2019, it was declared a web standard as it became the fourth language to run natively in web browsers, HTML, CSS, and JavaScript being the other three languages[8], [9], [10]. Another technology released in 1995 that will not be part of this study but is still worth mentioning is FutureSplash, which changed the name to Flash in 1996. Flash has

1 been a signiﬁcant part of the Web just as long as the Java applet, however, in 2017 Adobe announced that support for Flash would come to an end on December 31st 2020. The main reason for this is that newer technologies such as HTML5, WebGL, and WebAssembly are better options [11], [12]. While Java applets and ActiveX were originally created to produce a more interac- tive web, they remained popular because of their large increase in performance compared to JavaScript at the time. Asm.js, PNaCl, and WebAssembly all try to achieve native performance on the Web, it is, therefore, an interesting angle to compare the different technologies to one another to determine which one offers the best performance on the Web. While Java applets and ActiveX are very old technologies and are no longer recommended to use, there are still websites out there that use them. A reason why some companies might not have changed their use of technology could be because they already have a working product and do not see a reason to spend time redeveloping it if the other technologies offer the same performance. This study aims at highlighting the differences in performance between the technologies so that legacy applications can be updated using modern technologies.

1.2 Related work Finding research that compares the technologies to one another was difficult. Most performance tests were between JavaScript, WebAssembly, and native code; since WebAssem- bly’s main goal is to rival native performance on the Web. One article that mainly focuses on comparing WebAssembly to native performance does include a short performance comparison between WebAssembly and asm.js. Jangda Abhinav et al. [13] give a brief performance comparison between WebAssem- bly and asm.js in Google Chrome and Firefox web browsers. While the majority of the thesis focuses on comparing WebAssembly to native code, which is outside of this thesis’s scope, their comparison between WebAssembly and asm.js is relevant to this thesis’s research. In their thesis, they concluded that WebAssembly is faster than asm.js with a 1.54x mean speedup in Chrome and 1.39x in Firefox. At the end of their thesis, they include a short description of all but one (Java applets) of the technologies covered in this thesis and why WebAssembly is the preferred choice. For ActiveX, they mention that its unrestricted access to the user’s system was one of its downfalls. For PNaCl, they mention that, while it is an improvement over NaCl, it still "exposes compiler and/or platform-specific details such as the call stack layout." For asm.js, they mention that adding new features, such as 64-bit integers, would require extending JavaScript as a whole. Some improvements WebAssembly provide compared to asm.js that they mention are "(i) WebAssembly bi- naries are compact due to its lightweight representation compared to JavaScript source, (ii) WebAssembly is more straightforward to validate, (iii) WebAssembly provides formal guarantees of type safety and isolation, and (iv) WebAssembly has been shown to provide better performance than asm.js." In a non-scientific article, David Tippett [14] compares the performance between We- bAssembly and PNaCl using multi-threading when rendering PDF documents in Chrome. They concluded that WebAssembly is faster than PNaCl in certain situations but slower in others. Where WebAssembly proved to be faster was the first time a user viewed a document, it loaded their viewer library faster before the viewer was cached on the client, and it also proved to perform basic math operations faster than PNaCl. WebAssembly proved to be slower when rendering the PDF documents, however, by quite the margin. For their simple and moderately complex documents, PNaCl was 62% and 23% faster,

2 respectively, and for their large and complex documents, PNaCl proved to be 122% faster than WebAssembly. It is fascinating that PNaCl seems to be a lot faster than WebAssembly during heavy work, such as rendering PDF documents. If PNaCl is indeed as fast as the article claims compared to WebAssembly, it must mean that WebAssembly outperforms PNaCl in other areas by an equal or even more signiﬁcant margin.

1.3 Problem formulation Since the main objective of the technologies covered in this study is to increase the performance on the Web, it seems strange that WebAssembly is heavily recommended but is outperformed by PNaCl with a significant margin. If WebAssembly is, indeed, slower than PNaCl there is a possibility that WebAssembly is an improvement in other areas. One important area, when it comes to technologies that can be used on the Web, is security. If WebAssembly is slower but more secure, it could be enough to recommend it over other technologies. There have not been any studies comparing WebAssembly to older technologies such as Java applets and ActiveX, no studies that were found at least. Therefore, it is of interest for companies who still use those technologies to get a view of how they compare to newer technologies. If it turns out that Java applets or ActiveX outperform WebAssembly as well, then what is it that makes WebAssembly so great compared to the older technologies? Also, if it turns out that WebAssembly does not have greater performance than some of the other technologies, or at least does not heavily outperform them, the cost of migrating to WebAssembly might be too large to warrant the migration. This could be especially true for systems that are only used locally by companies and are not accessible to the open Web. If a system is only used locally, security might not be a considerable concern. If WebAssembly is the quickest technology, but only by a small margin, the migration to WebAssembly might take longer than the time it would save in execution time. Since no performance comparisons were found between ActiveX or Java applets to WebAssembly, it would be difficult for companies that use these old technologies to determine the efficiency of migrating to WebAssembly. One could argue that ActiveX and Java applets have been dead for years and should not be in use any longer and companies using either ActiveX or Java applets should have started migrating years ago. However, for some reason, some companies have decided not to migrate yet, which could potentially be contributed to one of the reasons mentioned above. The issue that this thesis, therefore, tries to solve is to provide these companies with the information needed to make a decision on whether or not it is worth migrating to newer technology, more specifically to WebAssembly. The problems this thesis aimed at solving were:

• Determine if PNaCl, or any of the other technologies, have better performance than WebAssembly and by what margin.

• If any of the other technologies outperform WebAssembly, determine in what other areas WebAssembly might be an improvement or if it lacks compared to the older technologies in other areas as well. The other areas included in this study are security and browser support.

3 1.4 Objectives The main objective of this thesis is to perform a controlled experiment that compares the performance of the technologies mentioned earlier. The goal of the controlled experiment is to establish a hierarchy between the technologies in terms of performance. To perform the experiment, a test environment was set up, and applications were developed using the technologies. While the performance experiment was the main focus, there was also a comparison in terms of security and browser support. The results of the performance and security comparisons are used to provide inside into whether or not it is worth migrating a legacy system to newer technology.

1.5 Scope/Limitation The technologies covered in this research are not all the technologies aimed at improving performance on the Web. Technologies such as Flash and Silverlight were left out because of time restrictions but would have been suitable candidates to include in this research.

1.6 Target group The main target group of this thesis is companies or developers that have existing implementations of systems using either ActiveX, Java applets, Asm.js, or PNaCl. These systems should be focused on performance since that is what this thesis studied. The companies or developers would have an interest in updating their current system by migrating to WebAssembly.

1.7 Outline The next chapter covers the methods used during the research. Chapter 3 provides an overview of the different technologies, more in-depth than the background, as well as mention some key aspects of performance testing. Chapter 4 covers how the data was collected. Chapter 5 covers the results of the performance tests and also explain more in-depth what the differences are between the different technologies. Chapter 6 gives an analysis of the results while Chapter 7 discusses the results in a more personal way. Chapter 8 ends the research with a conclusion.

4 2 Method

This chapter explains the methods used to solve the research questions from the Introduc- tion chapter.

2.1 Scientiﬁc Approach To answer the question of what WebAssembly does differently compared to the older technologies in terms of security, a case study was performed by gathering data through a qualitative literature study. The literature study was performed by searching for articles on Google Scholar, OneSearch, and when they came up short, through regular Google. To answer the question of which technology has the best performance, a quantitative controlled experiment was performed. Small applications were developed for each technology and then compared to one another through a series of performance benchmarks.

2.1.1 Literature study A literature study is performed by reading and synthesizing data from published articles within the area of the subject. By looking at what others have researched and comparing the results of that research to what others have concluded, it allows the one performing the literature study to draw conclusions on their own. Since the only data gathered in a literature study comes from what others have published, no "firsthand" data is collected. Articles are often collected from popular digital libraries, and the decision of which articles to include in the literature study is made by defining inclusion and exclusion criteria. Once the criteria are defined, the abstract section of the articles is put through the inclusion and exclusion criteria to determine if they are worth reading in their full text. In the case of this thesis, the literature study was used to determine differences between the technologies in terms of security. The literature study was performed by searching for articles on Google Scholar, OneSearch, and as a last resort when no articles were found; regular Google. If regular Google was used, the only source of information that was included was from articles written by people in the field of computer science or information gathered from the technologies’ official homepages and specifications. To determine whether an article seemed to contain information about the security of one of the technologies, the article’s abstract was read. If the abstract mentioned one of the technologies, the article was briefly read in its full text. The brief reading of an article was used to determine if it, indeed, contained information about the security for one of the technologies. If the brief reading proved successful, the article was read more thoroughly. The literature study does not give rise to any direct or indirect ethical considerations.

2.1.2 Controlled Experiment A controlled experiment is generally when a system is tested inside a controlled environment. The test produces quantitative data that is used to answer some research question. A controlled experiment consists of one or more dependent variable and independent variable. The independent variable is something that can be modiﬁed manually, and the dependent variable is the result of the test, which will differ depending on the independent variable. The controlled experiment of this study was used to compare the performance of the technologies covered in this study. The controlled experiment consists of four independent variables, technology, browser, application, and hardware. The dependent variables

5 are response time, throughput, and capacity. The response time is how long it takes for the web page to load, from the request to the server to the embedded application being ﬁnished. Throughput will be measured by the time it takes for the embedded application to ﬁnish performing its functions; this does not include the response time from the server. Capacity will be measured by including the CPU and RAM usage of the computer while the embedded application is performing its functions. In this thesis’s study, the results were calculated using the median value; this was to help eliminate any outlying values that could arise if some background process on the computer starts while the tests are running.

2.2 Reliability and Validity There are certain factors that could be considered to lessen the validity of the study. One such factor is that only devices running the Windows operating system are used when performing the tests. The reason for this is that ActiveX does not work on other operating systems and because there was no access to other devices at the time of this study. Also, the applications developed in this thesis are all small and not valuable in real-life situations. Therefore, there is a possibility that the results produced by the experiment included in this thesis will not be the same as one would get if running a large-scale real-life application. Another possible validity factor to consider is that background processes could affect the results of the performance tests. To try and decrease the risk of this happening, the choice of calculation (median) helps to remove the impact of outlying results. There were also measures taken to prevent background processes from running while performing the tests. Running the tests multiple times also reduce the impact background processes could have on the results. To help others reproduce the results from this study, the relevant hardware information is provided as well as the versions of any software used. To verify that the results of the experiment are different enough to be worth considering, the results were put through tests to see if they show statistically signiﬁcant difference. The programmer that developed the applications did not have any prior experience in developing applications using the technologies included in this thesis. Therefore, there is a possibility that the implementation of the applications could be faulty. However, to reduce the risk of any implementation faults occurring, ofﬁcial tutorials were used when developing the applications.

6 3 Dynamic web performance

This chapter gives a brief explanation of how web applications have evolved through the years and also explores if the technologies covered in this thesis can be used on the server-side of web development as well. This chapter also explains how the technologies function in a not too detailed fashion. The reasoning for not going too in-depth is that it would have consumed too much time to explore each technology in detail. In the end, there is also a general look at how performance testing can be done.

3.1 Dynamic web As mentioned earlier in Chapter 1, Java applets were the first successful attempt at making the Web more dynamic instead of static. Before Java applets, dynamic web pages were achieved by using server-side scripting. Server-side scripting entailed the browser sending a request to the server which modified the HTML page based on the request and then sent the HTML page back to the browser in the response. The communication between the browser and the server was most commonly achieved through the use of the Common Gateway Interface (CGI)[15]. Figure 3.1 aims at providing a visual representation of the flow of server-side scripting.

Figure 3.1: High-level ﬂow of a server-side scripting application.

The success of the applets resulted in JavaScript being created, and with the rise of JavaScript; it allowed for dynamic web pages to include client-side scripting. In client- side scripting it is the browser that updates the HTML DOM of the web page. The server in client-side scripting is mainly used to save and fetch data. Client-side scripting gave rise to a popular architecture in the shape of Single Page Applications (SPA). A web page that is a SPA never leaves the initial page. To simulate the changing of web pages, the browser updates the DOM through the use of JavaScript. This gives the web page a better user-experience and also makes it feel more similar to a desktop application[16]. Figures 3.1 and 3.2 together provides a visual difference between client-side and server- side scripting.

7 Figure 3.2: High-level ﬂow of a client-side scripting application.

While the Java applets and ActiveX include their own graphical interfaces, the newer technologies do not. The reason why the newer technologies do not include their own graphical interface is because JavaScript has evolved to the point of it being close to unbeatable when it comes to graphical interfaces on the Web. The focus of the newer technologies is instead honed in on performance, which is an area where JavaScript still falls short. Therefore, the newer technologies rely on JavaScript to handle most graphical components while the embedded technologies handle the heavy computational tasks.

3.2 Technologies server-side While this thesis only used ActiveX, Java applets, asm.js, PNaCl, and WebAssembly on the client-side, some of the technologies can also be used on the server-side. One example of an ActiveX control that is used as an HTTP/S server is "PowerTCP WebServer for ActiveX"1, it also has support for the SOAP protocol. Java applets do not natively run on the server-side, although it is possible to invoke an applet method from a server framework. The server-side equivalent of the Java applet is the Java servlet. Although servlets cannot handle a server on their own, they are used on top of an already existing server implementation; similar to how applets enhance the HTML page they are embedded on[17]. No uses of asm.js or PNaCl on the server-side could be found. While WebAssembly’s main use case is within a web browser, it can also be used outside of the browser[18]. WebAssembly can be used server-side with tools such as Wasmer2 and WASI3.

3.3 Technology overview The following section gives an overview of the different technologies covered in this study.

3.3.1 Java Applets Java applets work by pointing a HTML applet element, since HTML4 it is recommended to use the object element instead[19], to the applet’s Java class ﬁle. The class ﬁle contains bytecode which is executed by the Java Virtual Machine (JVM) of the user’s machine,

1PowerTCP WebServer for ActiveX 2Wasmer 3WASI

8 as can be seen in Figure 3.3. The JVM allows the Java applet to be accessible on any operating system that supports a JVM, which fulﬁls the "Write Once, Run Anywhere" goal of Java[20]. The JVM that runs the applet is separated from the browser on the operating system level. The applet loads in the background so that the web page remains responsive during loading. When the applet is ﬁnished loading, it appears on the web page and is ready to be interacted with. Unlike the JavaScript interpreter of the web browsers, which is single- threaded, the Java applet is multi-threaded. This is something that should be taken into consideration when developing applets that communicate with the JavaScript located on the web page[21]. Java applets are not widely used anymore, and with the release of Java 9 in 2017, the applets were deprecated[22]. Most websites that used Java applets are now switching to other alternatives. For example, the Massive Multiplayer Online Role-Playing Game (MMORPG) RuneScape removed their Java applet version of the game in December, 2019[23].

Figure 3.3: High-level ﬂow of a Java applet application.

3.3.2 ActiveX ActiveX controls are built using the Component Object Model (COM) speciﬁcation. COM was Microsoft’s attempt at making applications platform-independent. While it is possible to use languages such as C++ inside of an ActiveX control, its structure is built using COM. Data types deﬁned in COM are meant to be interpreted the same way no matter what machine or platform runs it[24, p. 319-324]. When developing ActiveX controls, there are two libraries available: the Microsoft Foundation Classes (MFC) and the ActiveX Template Library (ATL). As of writing this thesis, both libraries are still available in the newer versions of Visual Studio[24, p. 320]. ActiveX controls support asynchronous loading, which avoids blocking the single thread of the web browser. They can also be embedded without a user interface and can instead be used for faster calculations by invoking methods of the ActiveX control using the JavaScript code in the browser[25]. Microsoft stopped supporting ActiveX development in 2015 and are instead supporting the use of more modern technologies such as HTML5, JavaScript, and WebAssembly modules. A clearer showcase of this is that ActiveX can not run in Microsoft’s newer browser, Microsoft Edge[26]. ActiveX controls are embedded on a web page through the use of the HTML object element. The object element has a classid attribute that contains the id of the ActiveX control installed on the computer. If the control is not already installed, the object element

9 also contains a codebase attribute which points to the location of where to download the control from. The control gets installed on the user’s system and is executed with access to any part of the machine. A comparison between ActiveX and Java applet can be seen in Figures 3.3 and 3.4, which shows that the applet is contained within the JVM while the ActiveX control has full access to the machine.

Figure 3.4: High-level ﬂow of an ActiveX application.

3.3.3 Asm.js The specification of Asm.js describes the language as "a strict subset of the JavaScript language, providing a low-level, efficient target-language for compilers. Similarly to the C/C++ virtual machine, asm.js provides an abstraction through the use of a large binary heap with efficient loads and stores, integer and floating-point arithmetic, first-order function definitions, and function pointers"[6]. Unlike regular JavaScript, which uses Just-in-Time (JIT) compilers, asm.js can be compiled Ahead-of-Time (AOT). Compiling AOT provides performance benefits such as "unboxed representations of integers and floating-point numbers, absence of runtime type checks, absence of garbage collection, and efficient heap load and stores."[6] Code that fails to validate during the AOT compilation falls back to JIT compilation[6]. Asm.js code is not meant to be written by hand. Instead, it is meant to be written in other languages such as C++ and through tools such as Emscripten, compile into asm.js. Emscripten generates a JavaScript file containing all the "glue-code" necessary to use the asm.js module with regular JavaScript. Since asm.js is a subset of JavaScript, it is possible to run asm.js in any web browser. However, not all browsers have implemented the optimizations for asm.js which means it would be executed with the same performance as regular JavaScript. Asm.js code is included on a web page by pointing a script element to the JavaScript file containing the "glue-code" and the asm.js code. The asm.js functionality can be accessed outside of the asm.js file through the use of the Module object. Figure 3.5 show that, unlike ActiveX and the applet, asm.js is executed within the Web browser.

10 Figure 3.5: High-level ﬂow of an Asm.js application.

3.3.4 Portable Native Client Portable Native Client (PNaCl, pronounced "pinnacle") is an extension of Google’s Na- tive Client (NaCl). The main difference between PNaCl and NaCl is that PNaCl is architecture-independent. NaCl requires different implementations depending on which system architecture accesses the website containing the application. PNaCl solves the architecture dependency by compiling native code to a portable executable (pexe). The pexe file is served to the browser via a server, and before the code executes in the browser, the pexe file is translated to the appropriate native executable (nexe), as can be seen in Figure 3.6[27]. To compile native code, like C++, to a pexe requires the use of the NaCl SDK, which includes the Pepper Plug-in API (PPAPI). PPAPI makes it possible for C/C++ modules to communicate with the hosting browser, and it also grants access to system-level functions in a safe and portable fashion. For instance, PPAPI lets the NaCl module read and write to files, however it can only access files stored in Chrome’s sandboxed local disk[27]. PNaCl makes use of the HTML embed element when added to a website. The embed element points to a manifest file (.nmf) which contains different options as well as the location of the pexe file[27]. Currently, the pexe to nexe translator is only available on Google Chrome. PNaCl is also no longer recommended by Google; instead, Google recommends the use of WebAssembly[27].

Figure 3.6: High-level ﬂow of a Portable Native Client application.

11 3.3.5 WebAssembly WebAssembly is a "low-level assembly-like language with a compact binary format"[8]. Similarly to asm.js, WebAssembly is also a compile-target for other languages such as C, C++, Rust, and many more. Since WebAssembly is a binary format, it is not meant to be written by hand. However, WebAssembly does provide a textual format of the language called WebAssembly text format (WAT)[28]. As mentioned in Chapter 1, WebAssembly was recently made into a web standard as it runs natively in web browsers. This means that the same Virtual Machine located inside of the web browsers that load JavaScript code, can also load WebAssembly code, which can be seen in Figure 3.7. WebAssembly is based on a stack machine where sequences of instructions are executed in order. There are only four value types in WebAssembly, i32, i64, f32, and f64 which are 32 and 64 bit integer and float values. WebAssembly also consists of a linear- memory which is a large array of bytes. The linear memory has an initial size which can be increased dynamically as needed. There are also functions, tables, and modules in the WebAssembly specification. A function is as one would expect, a function takes some values and performs some operation and then returns values. The table is an array of function pointers that can be used to call functions indirectly. The module contains the definitions of the functions, tables, linear memories, and variables[29]. While the main motivation for creating WebAssembly was to run it on the Web, there is nothing that prevents it from running on other platforms[30]. As mentioned earlier in this chapter, WebAssembly can also be used server-side. Some other use-cases for We- bAssembly could be "Game distribution service (portable and secure), server-side compute of untrusted code, hybrid native apps on mobile devices, and symmetric computations across multiple nodes"[18]. WebAssembly is used on the Web by instantiating the module. If a start function was defined, the function would execute once the module is instantiated. If no start function was defined, it is possible to use the Module object with JavaScript to call exposed functions of the WebAssembly module.

Figure 3.7: High-level ﬂow of a WebAssembly application.

3.4 Performance Testing Performance testing can be described as the "comparison of one or more products to an industrial standard product over a series of performance metrics"[31]. Deﬁning when an application is performing well can be a challenging task because what performance classiﬁes as differ between types of applications. A web-based application might consider

12 good performance as quick response times from the server, while a single-player game might consider good performance as lower-end machines getting high frame rates while playing the game. Molyneaux, in their book, suggests that performance is a matter of perception and that "a well-performing application is one that lets the end user carry out a given task without undue perceived delay or irritation"[32, p. 1]. When measuring the performance of an application, there are certain key performance indicators (KPIs) that can be considered. KPIs can be divided into two types: service- oriented and efficiency-oriented. Service-oriented indicators include availability and response time, and they measure how well the application is providing its services to the end-user. Efficiency-oriented indicators include throughput and capacity, and they measure how well the application makes use of the hosting infrastructure[32, p. 2-3]. Once the performance tests have been completed, the results need to be calculated. There are multiple ways to do this; the first is to calculate the arithmetic mean or median values. The median has an advantage over the mean value in that if there are a few outlying values (e.g. 2, 3, 3, 19) the mean can become skewed while the median reflects the values more accurately. Another way of calculating the results is to use the standard deviation, the average variance from the mean value. It is based on the assumption that most data exhibit a normal distribution. If the standard deviation is a large value, it could indicate an erratic end-user experience where the results can vary by a significant magnitude. The final example is to use the nth percentile when selecting which values to include in the calculation. If only the 80th percentile of values were used in the previously mentioned values, it would include 2, 3, 3, which could then be used to calculate the arithmetic mean[32, p. 94-95].

13 4 Data Collection

This chapter explains how the data used in the comparison of the technologies was collected. It aims to provide the reader with the required information to be able to replicate the data collection in the most similar way possible.

4.1 Design The design phase of the data collection consisted of choosing which applications to use, what to use when collecting performance data, how the process of the performance data collection would work, as well as how to collect data on qualities outside of performance.

4.1.1 Design for collecting performance data The design of the performance data collection consisted of learning how to implement each technology on a web page. Once each technology was implemented, deciding on which applications to develop for the technologies came next. Deciding on what software to use to collect the performance data was done once the applications had been selected. After the data collection software had been established, the actual test process was decided.

Application selection Before considering which applications to develop for the performance tests, simple "Hello World" applications were developed for each technology to ﬁgure out how to embed each technology on a web page. All technologies except for ActiveX went without much trou- ble. The COM code of the ActiveX control was very confusing to someone who had never worked with COM code previously. There was no time to learn how to work with COM on an advanced level, which made it clear early on that the applications developed for the tests would not be very complex or include graphical components. While testing the graphical components would have been interesting, WebAssembly, asm.js, and PNaCl are all meant to be used alongside JavaScript, where JavaScript handles the graphical components. Therefore, to save time, it was considered an acceptable loss to not include graphical components in the tests. During the development of the Hello World applications, a simple server was also created to serve the ﬁles to the browser. Once the Hello World applications were up and running, the process of choosing what applications to develop for the tests began. Some criteria, listed below, were established to determine if an application was suitable or not.

• Does the application perform heavy-computational tasks?

• Is the application of an acceptable scale so that it can be developed within the given time-frame?

• Is the application possible to replicate for the technologies included in this study?

The applications that were selected were an algorithm that calculates the ﬁrst 43 numbers of the Fibonacci sequence by using recursion. A recursive Fibonacci algorithm is a good way of testing the technologies’ function call effectiveness. The second application selected was ﬁlling an array of length 30 million with random integers and then randomly

14 shuffling it, to test the memory management of the technologies. The third application selected was comparing 100 million pairs of random integers to one another, to test numeric computations. While these applications are not ones that would be used in actual systems, their aim is to provide insight into if some of the technologies are especially efficient at one or more specific areas of execution, for example, memory management. Because the applications are small, it allows for a very similar implementation of each application across the different technologies, which is both time-efficient and could potentially give more reliable results if the technologies execute very similar code. As mentioned earlier, the applications do not include any graphical components; this does stray from one of the primary use cases of ActiveX and Java applets since they became popular because they included their own graphical interface. While it would have been a valuable addition to the study to include an application with real-life use that makes use of the built-in graphical interface of ActiveX and Java applet, the time restraint of this study prevented that possibility. Also, it would have been challenging to implement the same application using asm.js, PNaCl, and WebAssembly because they do not have any built-in graphical interfaces. This could potentially reduce the reliability of the results if it turns out that, for example, Java applet is quicker than WebAssembly when executing the applications included in this study, but rendering and manipulating the graphical components of the applet is slower than rendering and manipulating HTML elements that WebAssembly makes use of. While all of the technologies, except for asm.js, support multi-threading, the applications did not make use of multi-threading. One reason for this is that handling multiple threads is more complex and takes longer to implement, and another reason is that these applications are very small and does not require the use of multiple threads. The implementation of the applications can be found in the GitHub repository available in Appendix 2.

Selecting performance data collection software When searching for a software to collect the performance data, it was determined that no publicly available performance testing software was available that fit this study’s re- quirements. It was, therefore, decided that an application would be developed, which could measure the execution times and hardware usage of the different technologies. To accomplish this, a timestamp is collected the moment before the browser sends a request to the server for the application. When the application has finished its function, another timestamp is collected and compared to the earlier one to get the load time of the page. The embedded application measures the execution time of its function internally, and the web page it is embedded in retrieves that data after the application has finished its function. ActiveX handles it slightly differently, instead of the web page retrieving the execution time from the embedded application; the ActiveX applications send the load time and execution time directly to the server via HTTP. Because some of the technologies can not automatically access the file system of the machine running it, the already created server was selected to help with that process. The server is a local Node.js4 server using the express5 framework. When the browser asks the server for the application, the server starts measuring the CPU and RAM usage of the system and then opens the application as a new process. The usage is measured using the

4Node.js 5Express

15 npm package systeminformation6 and the applications are started using the npm package open7. The server measures the computer’s total CPU and RAM usage, not just the applications. The reason why it includes all other processes running on the system is to ﬁnd the application’s process and then measure that speciﬁc process uses all of the computer’s processing power, which could obfuscate the results. The usage is measured every half of a second so that the server does not use too much of the system’s hardware when measuring the usage. Because the machine’s total CPU and RAM usage is measured, there has to be some way to make sure that the machine running the tests has the same baseline all the time while running the tests. To achieve this, all apps that are not essential will be shut down before running the tests. Examples of apps that should be turned off are Spotify, Slack, and Steam.

Design of the test process To be able to automate the execution of multiple sequential test runs of the applications, a base application was needed. The base application should handle the selection of application, technology, browser, and then execute the tests multiple times without the user needing to interact with the machine. Deciding how the flow of the performance tests was going to work was mainly decided by the base application needing to know when the application containing the technology finished its function. For the base application to know this, the server has to tell the client web page when a different web page has finished its function, which is not doable with regular HTTP request/response flow. It was, therefore, decided that the use of the WebSocket protocol was a good option to handle such functionality. The flow of the test process starts with the user pressing a button. The client collects a UNIX timestamp and sends the selected application, technology, browser, and timestamp to the server via WebSocket so that the server will later know which client to send the information to that the test finished. The server saves the timestamp in a JSON file, each technology has a dedicated JSON file and starts measuring the CPU and RAM usage of the machine. The server then uses the open package to start the selected browser at the proper URL for the selected application and technology. The newly created browser instance then runs the application which contains the selected technology version of the application. Once the application finishes its function, the client collects another UNIX timestamp and sends the execution time, timestamp, which application, and which technology that was used to the server via WebSocket. The server then stops measuring the machine’s usage and saves the timestamp, execution time, and usage in the same JSON file as mentioned earlier. Once the file is saved, the server tells the base application via WebSocket that the test finished and the server also sends data that the client can use to render graphs. If the tests have run the selected amount of times, then the base application renders the graphs and stops running any tests. If the tests have not run the selected amount of times, the test process starts over. The entire test process flow described above can also be seen in Figure 4.1.

6SystemInformation 7Open

16 Figure 4.1: Design of the test process ﬂow.

4.1.2 Design for collecting data on security To gather information for the case study, a literature study was used to synthesize information from published articles. There were no criteria defined on articles collected from either Google Scholar or OneSearch. The original plan had been to only include peer-reviewed articles; however, not many useful articles were found using those criteria which meant that regular Google was used more than Scholar and OneSearch. Therefore, it was decided that even articles that were not peer-reviewed would be included. There was, however, criteria set for regular Google, which is mentioned in Chapter 2. Only articles written by people in computer science would be included, or information gathered from official specifications of the technologies. This meant that information from sources such as forum comments would not be included. The literature study was performed during April and May 2020. The keywords shown in Table 4.1 were used when gathering information on the security of the technologies.

17 Keywords Search engine activex AND security Google Scholar & OneSearch applet AND security Google Scholar & OneSearch asm.js AND security Google Scholar & OneSearch pnacl AND security Google Scholar & OneSearch webassembly AND security Google Scholar & OneSearch +asm.js Google javascript AND security Google Scholar & OneSearch +webassembly Google +java +applet Google

Table 4.1: Keywords used during the literature study.

4.2 Performance experiment preparations To help others replicate this experiment, information on how to conﬁgure the browsers properly, and how to compile the technologies is discussed in this section.

4.2.1 Browser settings To run ActiveX controls and Java applets, the security settings in Internet Explorer need some modification. The easiest way of making sure that ActiveX controls and Java applets are allowed to run is by setting the browser’s security settings to at most Medium for the Local intranet zone. The browser’s security settings are accessed by clicking on the cog in the upper right corner of the screen and then selecting Internet Options. In the window that opens, select the Security settings tab. Next, select the Local intranet zone and then drag the slider so that the text next to it says Medium. One other measure that needs to be taken is to allow the applet to run. Allowing this is done by going into the Java settings and adding http://localhost:4000 to the Exception site list. To remove the popup message that appears when running a Java applet, set the Mixed code (sandboxed vs trusted) security verification setting under the Advanced tab of the Java settings to Disable verification. If this is not done, the user will manually have to allow the applet to run every time, which will obfuscate the load time results. NaCl is by default disabled in Chrome, to enable it navigate to chrome://flags, search for Native Client and enable it. Another browser setting that needs to be changed is allowing the applications to close themselves upon completion. By default this functionality is disabled in Firefox. To enable this functionality enter about:config in the URL search bar. On the config page, search for dom.allow_scripts_to_close_windows and set it to true.

4.2.2 Compilation tools All of the Listings mentioned in this section can be found in Appendix 1. Compiling C++ code to WebAssembly, asm.js, and PNaCl related ﬁles require third party compilation tools. Both WebAssembly and asm.js were compiled using Em- scripten8. The tests are run on Windows machines and to use Emscripten on Windows; Python9 needs to be installed. Next, the Emscripten SDK needs to be installed, and it is done by using the commands shown in Listing1 with a command prompt.

8Emscripten 9Python

18 Once the Emscripten SDK is installed, the C++ files are ready to be compiled into WebAssembly and asm.js files. To compile the C++ file, navigate to the folder containing the C++ file with the command prompt that installed the SDK. The commands for WebAssembly and asm.js compilation are almost identical; the only difference is the -s WASM flag, which is 1 when compiling to WebAssembly and 0 when compiling to asm.js. The -O3 flag specifies the amount of optimization that should be made -O3 is the most heavily optimized. The more optimization, the longer the compilation time but faster execution time. Listing2 shows the compilation command for compiling the array C++ file to WebAssembly. To use PNaCl, the NaCl SDK needs to be installed. The NaCl SDK requires Python as well. However, it needs to be Python version 2.7. Once Python is installed, the PATH environment variable needs to have a pointer to the Python location. To accomplish this, search for environment variables in the Windows search bar and click on the option that appears. Click on the Environment Variables... button. Under User variables, double- click on the variable path. Click the New button and enter the path to the Python version 2.7 folder. Next, download the NaCl SDK10 and unzip it in a chosen location. Once the NaCl SDK is unzipped, run the commands in Listing4 with a command prompt. The PNaCl files required for compilation consist of a C++ file, a Makefile, and a Make.bat file. The Makefile used for the tests in this thesis uses the same structure as the one used in Google’s "getting started" tutorial. The Make.bat file should contain the location to the make.exe file inside of the tools folder of the NaCl SDK. The content of a Makefile is shown in Listing3, and the make.bat content is shown in Listing5. Once the PNaCl files are set up correctly, the compilation process is ready to begin. In a command prompt, navigate to the folder containing the PNaCl files and run the commands shown in Listing6. To install the ActiveX controls, open the solution files located in the GitHub repository found in Appendix 2. Visual Studio 2017 was used to develop the ActiveX controls, and it is assumed that any attempts at replicating this experiment use the same software. The solution will not be able to build successfully without knowing where to find the curl library. The easiest way of letting Visual Studio know where to find curl is to download Microsoft’s C++ library manager, vcpkg11. Unzip the vcpkg into a chosen folder, open the Developer Command Prompt for VS 2017 and run the commands shown in Listing7. The ActiveX control solutions should now be able to build, Visual Studio should also be set to release mode and not debug mode.

4.3 Performance test execution How to execute the performance tests is described below.

4.3.1 Readying machine for testing Before starting any tests, all non-essential applications should be turned off. This was accomplished by opening the task manager and ending any task that was currently running that did not need to be running. Identifying which applications to turn off was done by restarting the machine and looking at which applications were running on boot, and then turning off any non-essential that had ﬂuctuating CPU and Memory usage. Once all the

10NaCl SDK 11Vcpkg

19 non-essential applications were turned off, a screenshot of the task manager was used as a comparison before each test run.

4.3.2 Running the tests The application that starts the tests was always executed from the Firefox browser. The main reason why Firefox was used instead of another browser was that when running the tests from Chrome, PNaCl consistently caused the browser to crash after 4-6 test runs, and the error could not be identified. Launching the test application is done by starting the local server and navigating to http://localhost:4000. The application’s user interface consists of one input, three selections, and one button. The input determines how many tests run to perform; the default is 30. The selections determine which application, technology, and browser to use. The button starts the tests, and while the tests are running, the button is disabled. While the tests are running, the machine should not be interacted with in any way. Once the tests are finished running, the results are displayed visually through multiple graphs. To view the graphs without running more tests, navigate to http://localhost:4000/graph, which brings up a similar interface to the test interface. On the graph page, the selections decide what app, and what metrics to compare. The graph page also contains an option to download the displayed graph as a png image file.

4.3.3 Hardware and Software The hardware used during the execution of the performance tests are described in Table 4.2.

Desktop Laptop Model Home-built Acer Aspire V3-772G Operating system Windows 10 Professional Windows 8.1 Processor Intel Core i7-8700k Intel Core i7-4702MQ Memory 16GB 8GB Graphics GeForce GTX 1080 GeForce GTX 760M Table 4.2: Speciﬁcation of the machines that the performance tests were executed on.

The software versions used during implementation and execution of the performance tests are described in Table 4.3.

20 Software Version Node.js 13.11.0 Express 4.17.1 Systeminformation 4.23.0 Open 7.0.2 Pepper API 49 Emscripten 1.39.6 Clang 10.0.0 Google Chrome 81.0.4044.92 Firefox 75.0 Microsoft Edge 81.0.416.58 Internet Explorer 11.719.18362.0 Table 4.3: The versions of the software used in this study.

21 5 Results

The data presented in this chapter is the data that was collected during the performance experiment and the literature study. The data from the performance experiment is presented through graphs, while the data from the literature study is presented through text.

5.1 Performance experiment This section contains the graphs obtained by running the performance tests 30 times per supported browser and then taking the median value of the results. Only WebAssembly (wasm) and asm.js were executed on multiple browsers since the other technologies only have support on one browser each.

5.1.1 Result explanation This section gives a short overview of what the different types of results imply. The execution time only measures the algorithm performed by the application, no browser or module loading is included in the execution time. The load time measures the time it takes for the browser to start and the module to load and ends when the application finishes its function. The CPU usage measures the total CPU usage of the machine, not just the application’s usage; the reasoning for this can be found in Chapter 4. The memory usage measures, just like the CPU usage, the machine’s total memory usage and not just the applications. Since WebAssembly and asm.js were executed on multiple browsers, Chrome, Firefox and Edge, their load time and execution time were measured as the median of all the browsers combined. However, Figures C.1-C.6 located in Appendix 3 does show the differences in execution time and load time of each technology per supported browser. The CPU and memory usage were calculated using the median value for each point in time during the loading of the technology. The usage is measured every 500ms, this means that the median is calculated by looking at the first 500ms and calculating the median of that per technology, and then move to the next 500ms and calculate the median of that. This process continued until the median of each point in time had been calculated. The length of the usage also had its median calculated instead of including the usage of every point in time. This was to prevent the usage of the test run that had taken the longest amount of time from not having any other values to compare within the final 500ms points.

5.1.2 Fibonacci application Here follows the results of the Fibonacci applications which calculated the 43 ﬁrst numbers in the Fibonacci sequence.

22 Figure 5.1: Execution times of the Fibonacci application.

Figure 5.2: Load times of the Fibonacci application.

23 Figure 5.3: CPU usage of the Fibonacci application running on the desktop.

Figure 5.4: CPU usage of the Fibonacci application running on the laptop.

24 Figure 5.5: Memory usage of the Fibonacci application running on the desktop.

Figure 5.6: Memory usage of the Fibonacci application running on the laptop.

5.1.3 Array application The Array application ﬁlled an array of length 30 million with random integers and then performed a random shufﬂe on the array.

25 Figure 5.7: Execution times of the Array application.

Figure 5.8: Load times of the Array application.

26 Figure 5.9: CPU usage of the Array application running on the desktop.

Figure 5.10: CPU usage of the Array application running on the laptop.

27 Figure 5.11: Memory usage of the Array application running on the desktop.

Figure 5.12: Memory usage of the Array application running on the laptop.

5.1.4 Numeric application The numeric application compared 100 million sets of random integers to each other.

28 Figure 5.13: Execution times of the Numeric application.

Figure 5.14: Load times of the Numeric application.

29 Figure 5.15: CPU usage of the Numeric application running on the desktop.

Figure 5.16: CPU usage of the Numeric application running on the laptop.

30 Figure 5.17: Memory usage of the Numeric application running on the desktop.

Figure 5.18: Memory usage of the Numeric application running on the laptop.

5.2 Qualitative results This section describes differences in the technologies that are not performance-based. The two areas that were identiﬁed that can differ between the technologies were security and browser support.

5.2.1 Security Security is a crucial aspect for any web page. If one of the technologies is embedded on a public web page, the users who visit the web page expect that page to be safe to use. Therefore, a literature study was performed in an attempt at discerning the differences in security between the technologies.

31 ActiveX ActiveX does not have a security model like the other technologies. Instead, ActiveX has a trust model; where the whole security responsibility is put onto the user. When the user accesses a website containing an ActiveX control, they are presented with an option of letting the control run its code or to block it. If the user agrees to let the control execute on their machine, the control gains full access to the machine. To prevent harmful controls of accidentally being run, Microsoft introduced digital certificates to assure users that controls are safe. These certificates are not foolproof as they can be stolen and used on harmful controls. Another attempt at preventing harmful controls from running is the security settings of Internet Explorer. If Internet Explorer is set to the highest security setting, no control without a certificate will be downloaded. The National Institute of Standards and Technology (NIST) assigned ActiveX the risk level of high[33], [34], [35].

Java applets Applets that are accessed from a Website are run in a sandbox. Certain applets that are signed by a trusted certiﬁcate can run outside of the sandbox. The sandbox restricts the access that an applet has to the client’s computer. Some of the things a sandboxed applet can not do are:

• If not launched through the Java Network Launch Protocol (JNLP), no access to the client’s local ﬁle system, executable ﬁles, system clipboard, or printers.

• Can not connect to or retrieve resources from any third-party server.

• Can not load native libraries.

• Can not change the SecurityManager.

• Can not create a ClassLoader.

• Can not read certain system properties.

Applets with a trusted certiﬁcate do not suffer from any of the restrictions mentioned above; they can run outside of the sandbox. If a trusted applet is accessed through JavaScript code, it is treated like a not trusted applet and will run inside of the sandbox. It should be stated that every applet requires the user’s permission to be run[36].

Asm.js Since asm.js is a subset of JavaScript and is executed as regular JavaScript; using a script tag, it beneﬁts from the same security principles as JavaScript does. JavaScript code executed in a web browser is, like the Java applets, run inside of a sandbox. The sandbox restricts the access that the JavaScript code has to the user’s system. The JavaScript code cannot read or write to any ﬁles without the user’s permission, nor can it load native code or libraries. Also, since JavaScript does not use pointers; determining the virtual address of JavaScript variables is impossible[37].

32 Portable Native Client It is not actually PNaCl that is executed in the browser; it is instead NaCl that is executed. NaCl has three different architectures that it can run on, x86-32, ARM, and x86-64, and they have slightly different implementations of security. The x86-32 architecture has both an inner-sandbox and an outer-sandbox. The outer- sandbox is not described in detail other than it works on the Operating System system-call level as an interceptor. The inner-sandbox works on the binary level, validating untrusted x86 code. The inner-sandbox uses static analysis when detecting security concerns within untrusted x86 code. Practices such as self-modifying code and overlapping instructions are not allowed in NaCl. Such practices are identified by reliably disassembling the code in a way that all reachable instructions are identified. The identified instructions can then be run through the inner-sandbox’s validator to verify that only legal instructions are present within the code. The inner-sandbox is meant to verify that any code that runs will not harm the user in any way[38]. The ARM architecture also makes use of the inner-sandbox to verify that no forbidden instructions can run. In addition to the security of the x86-32 architecture, ARM also verifies that untrusted code can not store to memory locations above 1GB, and can not jump to memory locations above 1GB[39]. The x86-64 architecture is similar to the ARM architecture except that instead of 1GB memory locations, x86-64 uses 4GB memory locations. x86-64 also designates a register within the 4GB memory location that is read-only to untrusted code[39]. The PNaCl sandboxing is similar to the JavaScript sandbox; it enforces the same- origin policy and keeps PNaCl separated from the local file system of the machine. PNaCl is, however, allowed a sandboxed file system that is non-persistent between application instances[40]. One security flaw of PNaCl that was found in the literature study was the possibility of a Prime+Probe attack. Since PNaCl uses the CPU of the machine it is executing on, it has access to the data cache. It is, therefore, possible to use memory-contention on the data cache to retrieve information of other processes running on the same CPU. Since PNaCl supports arrays, it makes it extra simple to perform memory-contention[40].

WebAssembly Just like Java applets, asm.js, and PNaCl; WebAssembly is also executed within a sandbox which contains the application so that it can not affect the rest of the machine. The only way for the application to access functionality outside of the sandbox is through safe and appropriate APIs. WebAssembly applications also execute deterministically, with very few exceptions12, which means that the application will behave in the same way each time it is executed[41]. During the development of a WebAssembly application, the developer can decide which functions to expose to the JavaScript code on the web page. Since only the selected functions are possible to call from JavaScript, potential attackers have limited access to the application. WebAssembly code is also immutable and impossible to observe at runtime. WebAssembly applications are, therefore, protected from control-ﬂow hijacking attacks, but they are not immune. There are still ways to hijack the control ﬂow of a WebAssembly application through code reuse attacks against indirect calls[41].

12Wasm determinism exceptions

33 WebAssembly applications are also immune to buffer overflow attacks, where adja- cent memory regions are accessed by exceeding the boundaries of an object. Local and global variables in WebAssembly applications are fixed-size and accessed by index and, therefore, safe from buffer overflows[41]. However, it is still possible to create buffer overflows within WebAssembly’s linear memory. This can give attackers access to local variables[42]. If a WebAssembly function that takes a parameter is called from JavaScript there is a possibility of an integer overflow. If the WebAssembly function is expecting a 32 bit integer but instead receives a larger number than a 32 bit integer can handle, it creates an integer overflow[42].

5.2.2 Browser support Knowing which browsers each technology can use is a major decision point when selecting technology. Table 5.1 presents the browser support for the technologies; the cells marked with a x indicate that the technology has support in the corresponding browser [43], [44], [45], [46].

ActiveX Asm.js Java applet PNaCl WebAssembly Google Chrome x x x Firefox x x Safari x Microsoft Edge x x Internet Explorer x x Table 5.1: Browser support for each technology.

34 6 Analysis

This chapter gives an in-depth analysis of the results presented in the previous chapter.

6.1 Performance experiment This section covers the different aspects of the performance tests and their results by analysing if the results have a statistically significant difference. To determine if the results have a statistically significant difference, the data used when extracting the median value, which is presented in the results, was also used to determine if the data samples exhibit a normal distribution. The Shapiro-Wilk test was used to determine if the samples exhibit a normal distribution. Since there are more than two samples in this experiment, the use of ANOVA was a requirement. The calculations were done by using the online application EZ Statistics13. If the samples did not exhibit a normal distribution, instead of ANOVA, the Kruskal-Wallis test was performed. Kruskal-Wallis determines if the data samples are statistically different by calculating a P-value. If the P-value was smaller than 0.05, there was a statistically significant difference. By running Dunn’s post-test, it could also be determined between which samples there was a statistically significant difference, and between which samples there was not. Figures 6.1- 6.12 show the data gathered during all the test runs. The data visualised in the box plots were used when determining if the results have any statistically significant difference. A box plot has five different summaries, the minimum value, the maximum value, the sample median, the first quartile, and the third quartile. Using Wasm as an example in Figure 6.1, the end of the line in the lower values indicate the minimum value while the end of the line in the higher values indicate the maximum value. The line inside of the box is the median value of the sample, the start of the box is the first quartile, and the end of the box is the third quartile.

6.1.1 Execution time This part analyses the results of the execution time for the different applications.

Fibonacci application Shapiro-Wilk’s test resulted in that no technology exhibited a normal distribution. Since none of the technologies exhibited a normal distribution, the Kruskal-Wallis test was used. The overall P-value returned by the Kruskal-Wallis test was very close to 0 because the P- value returned was 0.00000. Since the P-value was less than 0.05, there was a statistically signiﬁcant difference in execution time for the different technologies when running the Fibonacci application. Dunn’s post-test showed that only ActiveX - Java applet showed no statistically signiﬁcant difference. As can be seen in Figure 6.1, ActiveX, and Java applet have a more consistent execution time compared to the newer technologies. While asm.js and PNaCl have a few outlying execution times, they tend to have a rather consistent execution time considering their boxes are small. WebAssembly, however, does not seem to have as consistent execution time compared to the other technologies; at least not when there are many function calls involved. One could assume that the reason for WebAssembly’s wide range of execution times is because it is executed in multiple browsers. However, asm.js is executed in the same amount of browsers and does not have as wide a range of execution times.

13EZ Statistics

35 Figure 6.1: Box plot showing the variance of execution times for the different technologies running the Fibonacci application.

Array application None of the technologies exhibited a normal distribution according to the Shapiro-Wilk test. Since none of the samples exhibited a normal distribution, the Kruskal-Wallis test was used to determine if the technologies had a statistically significant difference. The overall P-value generated from the test was 0.00000 which meant it was very close to 0. Because the P-value was less than 0.05 it meant that there was a statistically significant difference. Dunn’s post-test resulted in that every combination of technology had a statistically significant difference. All technologies show a somewhat consistent execution time of the Array application, as evident by Figure 6.2. Asm.js seems to have the least consistent execution time, but the variance is minimal.

36 Figure 6.2: Box plot showing the variance of execution times for the different technologies running the array application.

Numeric application None of the technologies exhibited a normal distribution according to the Shapiro-Wilk test. Therefore, Kruskal-Wallis was used to determine the overall P-value. The overall P- value was very close to 0 once more because the test resulted in 0.00000, which meant that there was a statistically signiﬁcant difference in execution time between the technologies when running the numeric application. Dunn’s post-test found that all of the technologies showed a statistically signiﬁcant difference. The interesting result of Figure 6.2 is the variance of the asm.js execution time of the Numeric application. Figure C.5, which can be found in Appendix 3, tells the story of the large difference in execution time for asm.js. Numeric comparisons are better optimized on Firefox compared to Chrome, and Edge, which is the reason for the large variation in execution time.

37 Figure 6.3: Box plot showing the variance of execution times for the different technologies running the numeric application.

6.1.2 Load time This part analyses the results of the load time for the different applications.

Fibonacci application Only PNaCl exhibited a normal distribution according to the Shapiro-Wilk test, which meant that the Kruskal-Wallis test had to be used. The Kruskal-Wallis test, once more, resulted in the overall P-value being very close to 0 because it returned 0.00000, which meant that there was a statistically signiﬁcant difference in load time when running the Fibonacci application. Only ActiveX - Java applet showed no statistically signiﬁcant difference according to Dunn’s post-test. Figure 6.4 tells quite a clear story of how the tests were executed. Both asm.js and WebAssembly were executed in three different browsers (Chrome, Firefox, and Edge) which can result in different load times because the different browsers can take a different amount of time to start. Both ActiveX and Java applet was only executed in Internet Explorer and so their load time is quite consistent. PNaCl was only executed on Chrome but loading the PNaCl module was a very inconsistent process. As shown in Figure 6.4, PNaCl could be as fast as asm.js and WebAssembly, but for the most part, it took a long time to load the module.

38 Figure 6.4: Box plot showing the variance of load times for the different technologies running the Fibonacci application.

Array application Only PNaCl exhibited a normal distribution according to the Shapiro-Wilk test results. Therefore, instead of ANOVA; the Kruskal-Wallis test was used instead. The test resulted in an overall P-value of 0.00000 which meant that it was very close 0. Because the P- value was less than 0.05, the load time of the array application did have a statistically signiﬁcant difference. By running Dunn’s post-test, it was also determined that there was no statistically signiﬁcant difference between ActiveX and Wasm, as well as between Java applet and PNaCl. It is a very similar story for the Array application’s load time as to the Fibonacci application’s load time. WebAssembly’s execution time for the Array application was very consistent, as shown in Figure 6.2, but for the load time, it is more varied, as shown in Figure 6.5. Therefore, the assumption made for the Fibonacci load time that it is different browsers that have different startup times which is resulting in a more varied load time seems likely.

39 Figure 6.5: Box plot showing the variance of load times for the different technologies running the array application.

Numeric application Only PNaCl exhibited a normal distribution based on the results from the Shapiro-Wilk test. Since only one of the technologies exhibited a normal distribution, the overall P- value was calculated using the Kruskal-Wallis test. The overall P-value returned by the Kruskal-Wallis test was close to 0 because the returned P-value was 0.00000, which meant that there was a statistically signiﬁcant difference in load time between the technologies when running the numeric application. Dunn’s post-test showed that ActiveX - Java applet, ActiveX - asm.js, and Java applet - asm.js did not have a statistically signiﬁcant difference. Asm.js’s large variance in execution time, shown in Figure 6.3, can also be seen in its load time, Figure 6.6. One interesting thing, which is also true for the previous two applications’ load times, is that PNaCl is never faster than WebAssembly’s fastest load time. PNaCl consistently outperformed WebAssembly in execution time, but PNaCl’s fastest load time is never faster than WebAssembly’s fastest load time.

40 Figure 6.6: Box plot showing the variance of load times for the different technologies running the numeric application.

6.1.3 CPU usage This part analyses the results of the CPU usage for the different applications. The data samples for the CPU usage was extracted by calculating the mean CPU usage of each test run and then using those mean values as the sample data in the EZ Statistics application.

Fibonacci application Based on the results from the Shapiro-Wilk test, only PNaCl exhibited a normal distribution which meant that the Kruskal-Wallis test was used to determine the P-value. The overall P-value was close to 0 because it returned 0.00000, which meant that there was a statistically significant difference in CPU usage between the technologies when running the Fibonacci application. Dunn’s post-test showed that ActiveX - wasm, and asm.js - PNaCl showed no statistically significant difference. Figure 6.7 shows that the CPU usage of the technologies is very similar. PNaCl has a wider variance compared to the other technologies. A possible explanation for the wider variance could be that the CPU usage is very low when the PNaCl module is being fetched, as can be seen in the CPU usage figures of Chapter 5. However, even PNaCl’s variance is quite small, considering the difference between the minimum value and the maximum value is equivalent to approximately 10% of CPU usage, which is a small portion of CPU usage.

41 Figure 6.7: Box plot showing the variance of CPU usage for the different technologies running the Fibonacci application.

Array application None of the technologies exhibited a normal distribution according to the Shapiro-Wilk test. The Kruskal-Wallis test resulted in the overall P-value of 0.00000 which meant that it is very close to 0. Since the P-value was less than 0.05, there was a statistically significant difference in CPU usage when running the Array application. Dunn’s post-test showed that there was no statistically significant difference between a few of the technologies. ActiveX - Java applets, ActiveX - wasm, Java applet - asm.js, Java applet - PNaCl, and asm.js - PNaCl showed no statistically significant difference. In general, all the technologies used more CPU usage running the Array application, Figure 6.8, compared to the Fibonacci application, Figure 6.7. However, the difference is only in the 1-6% range, which is barely worth considering as a difference at all.

42 Figure 6.8: Box plot showing the variance of CPU usage for the different technologies running the array application.

Numeric application According to the results from the Shapiro-Wilk test, none of the technologies exhibits a normal distribution. Since none of the technologies exhibits a normal distribution, Kruskal-Wallis was used to determine the overall P-value. The overall P-value was 0.00000, which meant it was very close to 0. This meant that there is a statistically signiﬁcant difference in CPU usage between the technologies when running the numeric application. One thing that has been consistent for all the applications’ CPU usage is that ActiveX, and Java applet tend to have a smaller variance of usage compared to the other three technologies. This fact is made extra clear in Figure 6.9, where the ActiveX, and Java applet usage variance is very small.

43 Figure 6.9: Box plot showing the variance of CPU usage for the different technologies running the numeric application.

6.1.4 Memory usage This part analyses the results of the memory (RAM) usage for the different applications. The samples used for the memory usage were calculated similarly to the CPU usage by calculating the mean value of each test run. Similarly to the load time box plots, asm.js, and WebAssembly have a higher variance compared to the other three technologies, Figures 6.10- 6.12. The reason for this higher variance is most likely because the different browsers are using different amounts of memory.

Fibonacci application Only the Java applet exhibited a normal distribution according to the Shapiro-Wilk test, which meant that the Kruskal-Wallis test had to be used. The overall P-value returned from the Kruskal-Wallis test was 0.00016, which meant that there was a statistically significant difference in memory usage when running the Fibonacci application. Dunn’s post-test, however, showed that most of the technologies did not have a statistically significant difference. ActiveX - Java applet, ActiveX - asm.js, ActiveX - PNaCl, Java applet - asm.js, Java applet - PNaCl, and asm.js - PNaCl showed no statistically significant difference.

44 Figure 6.10: Box plot showing the variance of memory usage for the different technologies running the Fibonacci application.

Array application According to the Shapiro-Wilk test, only ActiveX exhibited a normal distribution. Since only one of the technologies exhibited a normal distribution, Kruskal-Wallis was used. The overall P-value returned from the Kruskal-Wallis test was 0.00000 or very close to 0, which meant that there was a statistically signiﬁcant difference in memory usage for the Array application. Dunn’s post-test showed that ActiveX - asm.js, and ActiveX - wasm did not have a statistically signiﬁcant difference.

45 Figure 6.11: Box plot showing the variance of memory usage for the different technologies running the array application.

Numeric application Only ActiveX exhibited a normal distribution according to the Shapiro-Wilk test. There- fore, the Kruskal-Wallis test was used to determine the overall P-value. The overall P- value was 0.00018, which meant that there was a statistically significant difference in memory usage between the technologies when running the numeric application. How- ever, Dunn’s post-test showed that most of the technologies did not have a statistically significant difference. ActiveX - asm.js, ActiveX - PNaCl, ActiveX - wasm, asm.js - PNaCl, asm.js - wasm, and PNaCl - wasm did not have a statistically significant difference.

46 Figure 6.12: Box plot showing the variance of memory usage for the different technologies running the numeric application.

6.2 Qualitative results Here follows the analysis of the qualitative results. The qualitative results include the security of the technologies as well as the browser support for each technology.

6.2.1 Security The focus of the security results was on what the technologies can and can not do when executed. If a technology is not allowed to do anything that could be used in a harmful way, then that technology would be very secure. ActiveX does not limit the code that executes in any way, shape, or form. If the user allows the ActiveX control to run, the control can do whatever it wants with the user’s machine. While ActiveX does include a verification message before running an ActiveX control to prevent possibly harmful controls from running, users could potentially click on the wrong option, and the control would execute. Java applets limit the access that an applet has when executed unless the applet is considered to be trustworthy. If an applet is trusted, then it is not restricted, it has the same access as a native Java application would have. Similarly to ActiveX, applets also include a verification message before executing, allowing the user to decide whether to run the applet. Asm.js has the benefit of being a subset of JavaScript, which means it has the same security as JavaScript does when executed in a browser. The JavaScript sandbox is quite limiting and heavily tested. PNaCl includes many security measures; it quarantines the application in a sandbox and also verifies that the code that executes is not harmful. The possible attack vector that the literature study found (Prime+Probe) is not necessarily a fault of PNaCl itself but

47 instead a fault of languages such as C/C++, where it is possible to access the memory regions where variables reside. WebAssembly is also sandboxed and only exposes specific points of the application to the JavaScript code running on the browser. WebAssembly seems to be immune to the Prime+Probe attack mentioned as a possible attack on PNaCl since WebAssembly is immune to buffer overflows. PNaCl is instead immune to code reuse attacks through its identification of instructions, while code reuse attacks are an attack that WebAssembly is open too.

6.2.2 Browser support Only the most popular browsers were included in the results for browser support. The results show that ActiveX and Java applets only have support on Internet Explorer. Asm.js has support on Google Chrome, Firefox, and Microsoft Edge. PNaCl only has support on Google Chrome. WebAssembly is the most supported technology with support on Google Chrome, Firefox, Safari, and Microsoft Edge.

48 7 Discussion

This chapter gives a more personal analysis of the results.

7.1 Execution time & Load time Based on the data presented in chapters 5 and 6, it becomes clear that PNaCl and We- bAssembly are overall the two technologies with the best performance, at least when looking at execution times. If the load time is also brought in to the discussion, then PNaCl trails far behind WebAssembly. While running the tests, it became apparent that PNaCl took a very long time to load the module most of the time. However, something that is not shown in chapter 5 is that PNaCl loads the module slightly faster when running the base application on Chrome. The problem with running the base application on Chrome, however, is that the PNaCl application causes the whole browser to crash after 4- 6 test runs. The crash occurs even when using simple "Hello World" applications, which is why the base application was run on Firefox instead. What is also interesting is that ActiveX and Java applet outperformed both PNaCl and WebAssembly for the Fibonacci application, in both execution time and load time. This could be an interesting use case for applications that use much recursion. Based on the results from the array application, ActiveX seems to have much better memory management compared to Java applets since it is almost twice as fast. Asm.js was consistently among the slowest technologies for each application. Performance-wise, asm.js can not be recommended over WebAssembly. The performance tests were all run on two different machines, this was used to determine if the performance of the technologies show different tendencies depending on the hardware. However, based on the results of Chapter 5, the tendencies seems to be the same for both the desktop and the laptop when it comes to execution time and load time. The previous research mentioned in chapter 1.2 seems to align somewhat with the results gathered in this study. The results of Jangda Abhinav et al. [13] research claims that WebAssembly is 1.54x faster in Chrome and 1.39x faster in Firefox compared to asm.js. Based on the Figures located in Appendix 3, this thesis found that for the Fibonacci application, WebAssembly was 1.14x faster on Chrome and 1.46x faster on Firefox. For the Array application, WebAssembly was 3.43x faster on Chrome and 3.89x faster on Fire- fox. Lastly, for the Numeric application, WebAssembly was 8.04x faster on Chrome and 3.55x faster on Firefox. This shows that Firefox has much better optimization for asm.js when it comes to comparing integers than Chrome has. While both this thesis and Janga Abhinav et al. conclude that WebAssembly is faster than asm.js, this thesis has found cases where WebAssembly is much faster than asm.js. As for the research that David Tippet [14] did regarding the performance between WebAssembly and PNaCl, their thesis found certain cases where WebAssembly outperformed PNaCl. Tippet mentions that WebAssembly is faster than PNaCl the ﬁrst time their library viewer is loaded before it is cached on the client. WebAssembly being faster the ﬁrst time the library viewer is loaded does make sense if Tippet includes the full load time since PNaCl takes time to load the module. For the actual loading of their PDF documents, PNaCl proved to consistently outperform WebAssembly, which is in line with the results of this thesis. The results answer which technology has the best performance and provides three different application cases that companies can consider when deciding if it is worth changing to a different technology. If all one cares about is the execution time, then PNaCl is the best option. If one considers the load time of the whole page and the execution time, then

49 WebAssembly is the recommended technology based on the performance results of this study.

7.2 CPU & RAM Usage The CPU usage is slightly skewed at the moment where at the very start, the usage either starts high and then drops or has a significant spike at the very start. The reason for this spike is that the usage starts measuring before the browser opens, which means that the browser opening is part of the usage. It is difficult to make anything out of the usage of the technologies; most of the time, they are very similar. While there is an overall statistically significant difference in both CPU and memory usage, the difference in usage between the technologies is minimal. While it seems that ActiveX and Java applets use less memory than the other three, it is only by a few hundred MB of RAM which is not much in today’s standards. Based on the CPU usage results, PNaCl consistently use less CPU compared to the other technologies. One reason for this could be that it has a long startup time and during that time it uses less CPU. WebAssembly and asm.js tend to use the most memory of the technologies when running on the desktop. Nevertheless, as mentioned earlier, the differences are minimal. The usage of the technologies is included, and if a company cares about the usage of their applications, then it can be useful information. As it is now, the difference in usage between the technologies is so small that it is not worth considering when choosing technology.

7.3 Security While this study does not go into too many details regarding the security of the technologies, it does give an overview of what the technologies’ restrictions are. ActiveX is the least secure of the technologies, due to that it does not restrict the application’s access to the user’s machine at all. The verification message is a weak security measure, especially since it can be turned off, as proven by this study. Putting all the responsibility on the user also seems like a very poor design choice since users are bound to click "accept" on any confirmation message presented on the Web to gain access to a website. While the idea of ActiveX is very intriguing, its security concerns do not make it suitable for today’s websites. The Java applet is an improvement over ActiveX but still suffers from similar issues to ActiveX. While a nontrusted applet is restricted in its access to the user’s machine, a trusted applet is not. As mentioned in the results for ActiveX, it is possible to steal the trusted signatures and include them on harmful applications. This is something that can affect applets as well as ActiveX. Applets also include a confirmation message where users have to allow the application to run, while these can also be turned off; it is not as easy to turn them off as it is for ActiveX. If a user turned off the applet confirmation message, it was not by mistake. No articles that discussed asm.js security could be found, however since asm.js is a subset of JavaScript and is executed in the same way as JavaScript, it also adheres to the same restrictions as JavaScript. Therefore, security concerns regarding asm.js are more focused on the data that the user enters. Usual attacks that can affect JavaScript are Cross- Site Scripting (XSS) and Cross-Site Request Forgery (CSRF). PNaCl and WebAssembly both seem to be quite secure. The attack vectors that were found for the technologies are, ironically, fixed by the other technology. WebAssembly

50 is immune to Prime+Probe attacks because it cannot get buffer overﬂows, and PNaCl is immune to code reuse because its disassembling of the code veriﬁes that no such code exists in the application.

7.4 Browser support Browser support can be a major deciding factor for users when it comes to deciding whether to use an application or not. If a user’s main browser does not support the technology used in an application, the user might decide that the hassle of changing the browser to use the application is not worth the gain of using the application. Based on the results, WebAssembly has the best browser support where all of the major browsers support it. It is only Internet Explorer that does not support WebAssem- bly, but in today’s age, Internet Explorer is no longer one of the most commonly used browsers. One could argue that asm.js has better browser support than WebAssembly since it is JavaScript. This means that the asm.js code will run on any browser that support JavaScript but it will only see increased performance on Chrome, Firefox, and Edge. Leading up to this point, WebAssembly and PNaCl have been relatively even, with PNaCl possibly being ahead of WebAssembly. However, the fact that PNaCl only has support on Google Chrome is a major disadvantage. While Chrome is, as of today, the most commonly used browser by quite the margin, only having support on Chrome loses approximately 38% of all users[47].

7.5 Summary The problem that this thesis aimed at solving was whether WebAssembly had better performance than other popular technologies. The experiment performed during this study shows that PNaCl has a faster execution time than WebAssembly for all three applications used during the performance tests. However, PNaCl has an overall slower load time than WebAssembly by a more significant margin than the difference in execution time. One could, however, argue that the execution time is more important than the load time. The reason for the execution time being more important is that the application only needs to load once while it can be executed multiple times during a session. Also, the applications developed during this study were all small, redundant applications, and a large-scale application could take much longer to execute its function, in which case, a technology with a faster execution time would be more beneficial than technology with faster load time. Both ActiveX and Java applet outperformed PNaCl and WebAssembly when running the Fibonacci application. However, the difference was marginal, and both PNaCl and WebAssembly beat ActiveX and Java applet by a much more significant margin when running the other two applications. The differences in CPU and RAM usage were slim and should not be considered when determining which technology performs the best. Therefore, the experiment conducted in this study shows that PNaCl has the overall best performance. However, the other part of the problem that this thesis aims at answering is whether WebAssembly is an improvement in other areas compared to the other technologies. Both WebAssembly and PNaCl implement several different security enhancements, and it is difficult to say that one is more secure than the other. However, WebAssembly has much better web browser support than PNaCl, which is a crucial factor for real-life applications. Also, PNaCl is not enabled by default in the Chrome browser, which means that even a large amount of Chrome users might not be able to use an application that implements PNaCl. PNaCl is also deprecated and will not see any improvements in the future. Therefore, it is possible

51 that browsers could achieve better optimization for WebAssembly in the future, which could push WebAssembly’s execution time past PNaCl’s. Since the tests were run 30 times for each supported browser (excluding Safari) on both a desktop and a laptop and both hardware show the same tendencies, the validity of the results is quite high. The applications were also developed to be as similar as possible in their execution and structure. However, as mentioned earlier, the applications used when running the tests are very small and have no real-life use. This could potentially affect the external validity of the results. It is possible that a large-scale real-life application could experience different results. Also, the tests were exclusively executed on the Win- dows operating system, which means that other operating systems could show different results. The results shown in Chapter 5.1 do not differentiate between browsers, meaning that for WebAssembly and asm.js, their results combine tests executed on Chrome, Firefox, and Edge. One could argue that combining the results of the browsers is a reliability risk since, as mentioned earlier in Chapter 6.1, different browsers can have different results. This is especially true for the Numeric application and asm.js where it is a lot faster on Firefox compared to Chrome and Edge. The way the results are calculated in Chapter 5.1 hides the fact that Firefox is quite quick for the Numeric application and asm.js while Chrome and Edge are quite slow. Therefore, since Chrome and Edge are equivalent to two-thirds of the test runs for the Numeric application, the median value will end up being from either Chrome’s or Edge’s run times. However, the idea for the results in Chapter 5.1 was to provide the overall performance of the technologies. Therefore, it makes sense to combine the results of multiple browsers when multiple browsers support the technology. One could even argue that because of Chrome’s massive user base, compared to Firefox and Edge, the results of Chrome’s test runs should weigh heavier than those of Firefox and Edge. However, if whoever reads this study is interested in the different execution times and load times per browser, Figures C.1- C.6, found in Appendix 3, do show those results. The validity of the security results is also high since only information gathered from published scientific articles and the technologies’ official specifications were used.

52 8 Conclusions & Future work

The goal of this thesis was to determine if WebAssembly is an improvement compared to the older technologies and, if so, also explore in what ways it is an improvement. The motivation behind this was for companies that still use older technologies to get a comparison between their technology and newer technologies to ease the decision making regarding the value of migrating to newer technologies. Since the main purpose of the technologies is to increase the performance on the Web, the main focus of this study was to compare the performance of the technologies. The performance test results show that PNaCl has the best overall execution time, with WebAssembly being slightly slower. If both load time and execution times are considered, WebAssembly is the best performing technology. The results also show that there is no noticeable difference in CPU or RAM usage between the technologies. PNaCl’s results are, in some ways, questionable, as mentioned in chapter 7; PNaCl loaded the module much faster when the base application was running on Chrome. If the crash can be ﬁxed, PNaCl’s load time can likely rival WebAssembly’s. While the tests were performed on two machines, one desktop, and one laptop, both machines were using Windows operating system. Therefore, it can be of interest to include tests from machines using other operating systems, if so, ActiveX will need to be removed from the tests since it requires the Windows operating system. ActiveX has the worst security of all the technologies. While Java applets are slightly more secure than ActiveX, they still suffer from the possibility of an applet gaining full access to a user’s machine. Based on these security concerns, neither ActiveX nor Java applets are suited for the open Web. Asm.js, PNaCl, and WebAssembly all seem to be quite secure, with only a handful of security concerns found. If browser support is a signiﬁcant concern, then WebAssembly is a clear choice. While asm.js also has relatively good browser support, it is heavily outperformed by WebAssem- bly. ActiveX and Java applets can not be recommended based on their browser support. PNaCl is questionable; Chrome is by far the most commonly used browser, and as such, PNaCl does reach the majority of users but locks out a large part of users that do not use Chrome. WebAssembly and PNaCl proved to be the overall "best" technologies of the ones covered in this study. This conclusion is based on the fact that they heavily outperform asm.js in the controlled experiment and have much better security than ActiveX and Java applets, and also heavily outperform ActiveX and Java applets when running the Array and Numeric applications. However, both the Java applet and ActiveX outperform We- bAssembly and PNaCl when running the Fibonacci application. Therefore, if a company has a Java applet system that mostly uses recursion, and is only located on a local server where security is not important, based on the performance, migrating to WebAssembly cannot be recommended. One question that arises is would it be worth migrating to Java applet from, for example, asm.js if the system uses a lot of recursions? If the answer is only based on the results of this study, and the system is not planned to be used on the open Web, there is an argument for it. However, the main purpose of this study was to help companies determine the worth of migrating to WebAssembly. The purpose of this study is not to prove the worth of migrating to older technologies; in fact, the reason behind this study is for companies to move away from older technologies. Therefore, companies should never consider migrating to old technologies such as Java applets or ActiveX. In conclusion, if a company has a system using either ActiveX or Java applet that is available on the open Web, it is worth migrating to WebAssembly based on security

53 concerns alone. However, if the system is only used locally and mostly uses recursion, a migration to WebAssembly might not be warranted. Otherwise, migrating from a system using ActiveX or Java applet to WebAssembly is, most definitely, recommended. If a company has a system using asm.js, it is also worth migrating to WebAssembly based on the difference in performance. Also, if the system is compiled using Emscripten, there might not need to be many code changes needed to migrate from asm.js to We- bAssembly. If a company has a system using PNaCl, it is not as easy to recommend a migration to WebAssembly. PNaCl outperformed WebAssembly in execution time for all three applications, and both PNaCl and WebAssembly are quite secure. However, as mentioned in Chapter 7.5, PNaCl is deprecated and will not see any future improvements while WebAssembly is strongly supported and will most likely see future optimizations. Therefore, it is likely that WebAssembly will be a better option compared to PNaCl in the future. Also, the browser support of WebAssembly is much better than it is for PNaCl. Based on the results of this study, it is difficult to recommend a migration from PNaCl to WebAssembly. As a final note, the recommendations in the paragraph above are only based on the results of this study. The overall consensus of companies and developers across the world is that WebAssembly should be used instead of the other technologies; this is especially true for ActiveX and Java applets. This study only aims to enforce this viewpoint while being as objective as possible when discussing the results. Future work is something that was mentioned earlier in this chapter. Summarizing the potential future work that can be done in this area of the subject is to perform the same tests, except for ActiveX, on other operating systems, which could be of value to companies. The three applications used in this study are very arbitrary and include no graphical components. Therefore, it can be of interest to do the same performance tests on more large-scale applications; to determine if it makes a difference. The technologies used in this study can theoretically be replaced with any other similar technologies. There is a possibility that other technologies, not covered in this study, could outperform We- bAssembly. Testing applications that make use of multi-threading is also an option for future work.

54 References

[1] W3Techs, “Usage statistics of JavaScript as client-side programming language on websites,” 2020, [Accessed: 2020-02-18]. [Online]. Available: https: //w3techs.com/technologies/details/cp-javascript

[2] M. Byrne, “The Rise and Fall of the Java Applet: Creative Coding’s Awkward Little Square - Motherboard,” 2016, [Accessed: 2020-02-19]. [Online]. Available: https://www.vice.com/en_us/article/8q8n3k/a-brief-history-of-the-java-applet

[3] C. Severance, “JavaScript: Designing a Language in 10 Days,” Computer, vol. 45, no. 2, pp. 7–8, feb 2012. [Online]. Available: https://www.computer.org/csdl/ magazine/co/2012/02/mco2012020007/13rRUy08MzA

[4] “Microsoft Announces ActiveX Technologies,” 1996, [Accessed: 2020-02- 21]. [Online]. Available: https://web.archive.org/web/20120201133255/http://www. microsoft.com/presspass/press/1996/mar96/activxpr.mspx

[5] B. Evans, Java: The Legend, ﬁrst edit ed., N. Barber, Ed. O’Reilly Media, Inc., 2015.

[6] D. Herman, L. Wagner, and A. Zakai, “asm.js,” [Accessed: 2020-03-03]. [Online]. Available: http://asmjs.org/spec/latest/

[7] A. Donovan, R. Muth, B. Chen, and D. Sehr, “PNaCl : Portable Native Client Executables,” Tech. Rep., 2010. [Online]. Available: https://css.csail.mit.edu/6.858/ 2012/readings/pnacl.pdf

[8] Mozilla Developer Network (MDN), “WebAssembly | MDN,” 2018, [Ac- cessed: 2020-03-03]. [Online]. Available: https://developer.mozilla.org/en-US/ docs/WebAssembly

[9] Webassembly, “Roadmap - WebAssembly,” [Accessed: 2020-03-03]. [Online]. Available: http://webassembly.org/roadmap/

[10] A. van der Hiel, “World Wide Web Consortium (W3C) brings a new language to the Web as WebAssembly becomes a W3C Recommendation,” 2019, [Accessed: 2020- 03-03]. [Online]. Available: https://www.w3.org/2019/12/pressrelease-wasm-rec. html.en

[11] Adobe, “Adobe Flash Player End of Life,” [Accessed: 2020-06-29]. [Online]. Available: https://www.adobe.com/products/ﬂashplayer/end-of-life.html

[12] J. Hoffman, “Flash And Its History On The Web - The History of the Web,” 2017, [Accessed: 2020-06-29]. [Online]. Available: https://thehistoryoftheweb. com/the-story-of-ﬂash/

[13] A. Jangda, B. Powers, E. D. Berger, and A. Guha, “Not So Fast: Analyzing the Performance of WebAssembly vs. Native Code,” in 2019 USENIX Annual Technical Conference (USENIX ATC 19). Renton, WA: USENIX Association, jul 2019, pp. 107–120. [Online]. Available: https: //www.usenix.org/conference/atc19/presentation/jangda

55 [14] D. Tippett, “WebAssembly vs. PNaCl Performance Benchmark | PDFTron,” 2019, [Accessed: 2020-03-04]. [Online]. Available: https://www.pdftron.com/blog/wasm/ wasm-vs-pnacl/

[15] T. Heninger and R. Rasmussen, “Server-side scripting language and programming tool,” Oct. 22 2002, uS Patent 6,470,349.

[16] S. Deshmukh, D. Mane, and A. Retawade, “Building a Single Page Application Web Front-end for E-Learning site,” Tech. Rep., 2019. [Online]. Available: https://ieeexplore-ieee-org.proxy.lnu.se/stamp/stamp.jsp?tp=&arnumber=8819703

[17] R. Mordani, “Java TM Servlet Specification F I N A L,” Tech. Rep., 2009. [On- line]. Available: https://s3.amazonaws.com/academia.edu.documents/32541681/ servlet-3_0-final-spec.pdf?response-content-disposition=inline%3Bfilename% 3DServlet_Specification_Version_3.0_F_I_N.pdf&X-Amz-Algorithm= AWS4-HMAC-SHA256&X-Amz-Credential=ASIATUSBJ6BALF2ELVHA% 2F2020

[18] WebAssembly, “Use Cases - WebAssembly,” [Accessed: 2020-05-18]. [Online]. Available: https://webassembly.org/docs/use-cases/

[19] W3C, “Objects, Images, and Applets in HTML documents,” 1999, [Accessed: 2020- 04-14]. [Online]. Available: https://www.w3.org/TR/html401/struct/objects.html

[20] L. Lemay, C. L. Perkins, and M. Morrison, Teach Yourself Java in 21 Days Profes- sional Reference Edition, C. Morrow, Ed. Taber, Mark, 1996.

[21] Oracle, “Applet Development Guide,” [Accessed: 2020-04-14]. [Online]. Available: https://docs.oracle.com/javase/8/docs/technotes/guides/deploy/applet_ dev_guide.html#A998882

[22] ——, “JDK 9 and the Java Plugin,” [Accessed: 2020-04-14]. [Online]. Available: https://java.com/en/download/faq/jdk9_plugin.xml

[23] Jagex, “Java has been sunset today - News - RuneScape,” [Ac- cessed: 2020-04-14]. [Online]. Available: https://secure.runescape.com/m= news/java-has-been-sunset-today

[24] R. Bitter, T. Mohiuddin, and M. Nawrocki, LabVIEW TM Advanced Programming Techniques, 2nd ed. CRC Press, 2000.

[25] Microsoft, “ActiveX Controls on the Internet | Microsoft Docs,” [Ac- cessed: 2020-04-14]. [Online]. Available: https://docs.microsoft.com/en-us/ cpp/mfc/activex-controls-on-the-internet?view=vs-2019

[26] ——, “A break from the past, part 2: Saying goodbye to ActiveX, VBScript, attachEvent. . . ,” [Accessed: 2020-04- 14]. [Online]. Available: https://blogs.windows.com/msedgedev/2015/05/06/ a-break-from-the-past-part-2-saying-goodbye-to-activex-vbscript-attachevent/

[27] GoogleChrome, “Technical Overview - Google Chrome,” 2018, [Accessed: 2020- 04-16]. [Online]. Available: https://developer.chrome.com/native-client/overview

56 [28] C. Watt, “Mechanising and verifying the webassembly speciﬁcation,” in Proceedings of the 7th ACM SIGPLAN International Conference on Certiﬁed Programs and Proofs, ser. CPP 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 53–65. [Online]. Available: https: //doi.org/10.1145/3167082

[29] WebAssembly, “Overview — WebAssembly 1.1,” [Accessed: 2020-05-19]. [Online]. Available: https://webassembly.github.io/spec/core/intro/overview.html

[30] A. Haas, A. Rossberg, D. L. Schuff, B. L. Titzer, M. Holman, D. Gohman, L. Wag- ner, A. Zakai, and J. Bastien, “Bringing the web up to speed with WebAssembly,” ACM SIGPLAN Notices, vol. 52, no. 6, pp. 185–200, sep 2017.

[31] V. Beiranvand, W. Hare, and Y. Lucet, “Best practices for comparing optimization algorithms,” Optimization and Engineering, vol. 18, no. 4, pp. 815–848, dec 2017. [Online]. Available: https://link.springer.com/article/10.1007/s11081-017-9366-1

[32] I. Molyneaux, The Art of Application Performance Testing, 2nd ed., O. Andy and B. Anderson, Eds. O’Reilly Media, Inc., 2009.

[33] J. M. Marek, “Plain English: Risks of Java Applets and Microsoft ActiveX Controls,” Tech. Rep., 2002. [Online]. Available: http://citeseerx.ist.psu.edu/ viewdoc/download?doi=10.1.1.602.639&rep=rep1&type=pdf

[34] S. Dugan, “Exposing the activex security model.” InfoWorld, vol. 19, no. 20, p. 98, 1997. [Online]. Available: http://search.ebscohost.com.proxy.lnu.se/login. aspx?direct=true&db=buh&AN=9706046293&site=ehost-live

[35] S. Garﬁnkel and G. Spafford, Web security & commerce. O’reilly Cambridge, MA, 1997.

[36] Oracle, “What Applets Can and Cannot Do (The JavaTM Tutorials > Deployment > Java Applets),” 2015, [Accessed: 2020-03-24]. [Online]. Available: https: //docs.oracle.com/javase/tutorial/deployment/applet/security.html

[37] Y. Oren, V. P. Kemerlis, S. Sethumadhavan, and A. D. Keromytis, “The spy in the sandbox: Practical cache attacks in javascript and their implications,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 1406–1418. [Online]. Available: https://doi.org/10.1145/2810103.2813708

[38] B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar, “Native client: A sandbox for portable, untrusted x86 native code,” in 2009 30th IEEE Symposium on Security and Privacy, May 2009, pp. 79–93.

[39] D. Sehr, R. Muth, C. Bifﬂe, V. Khimenko, E. Pasko, K. Schimpf, B. Yee, and B. Chen, “Adapting software fault isolation to contemporary CPU architectures,” Tech. Rep., 2010. [Online]. Available: https://www.usenix.org/legacy/events/sec10/ tech/full_papers/Sehr.pdf

[40] D. Genkin, L. Pachmanov, E. Tromer, and Y. Yarom, “Browser Security with a Grain of Salt: Drive-by Key-Extraction Cache Attack via PNaCl,” Tech. Rep.,

57 2017. [Online]. Available: https://www.cs.tau.ac.il/$\sim$tromer/drivebycache/ drivebycache-20170822.pdf

[41] WebAssembly, “Security - WebAssembly,” [Accessed: 2020-05-12]. [Online]. Available: http://webassembly.org/docs/security/

[42] B. Mcfadden, T. Lukasiewicz, J. Dileo, and J. En- gler, “Security Chasms of WASM,” Tech. Rep., 2018. [On- line]. Available: https://paper.bobylive.com/Meeting_Papers/BlackHat/ USA-2018/us-18-Lukasiewicz-WebAssembly-A-New-World-of-Native_ Exploits-On-The-Web-wp.pdf

[43] E. McManaway, “What Browsers Support ActiveX?” [Accessed: 2020-06-29]. [Online]. Available: https://www.techwalla.com/articles/ what-browsers-support-activex

[44] S. Melendez, “List of Web Browsers That Support Java Applets,” 2018, [Accessed: 2020-06-29]. [Online]. Available: https://www.techwalla.com/articles/ list-of-web-browsers-that-support-java-applets

[45] CanIUse, “Can I use... Support tables for HTML5, CSS3, etc,” [Accessed: 2020-06-29]. [Online]. Available: https://caniuse.com/#search=asm

[46] Google, “NaCl and PNaCl,” [Accessed: 2020-02-26]. [Online]. Available: https://developer.chrome.com/native-client/nacl-and-pnacl

[47] W3Counter, “W3Counter: Global Web Stats,” [Accessed: 2020-05-13]. [Online]. Available: https://www.w3counter.com/globalstats.php

58 A Appendix 1 git clone https://github.com/emscripten-core/emsdk.git cd emsdk git pull emsdk install latest emsdk activate latest emsdk_env.bat Listing 1: Installing the Emscripten SDK emcc -O3 array.cpp -s WASM=1 -o array.js -s "EXPORTED_FUNCTIONS=[’ _getExecTime’,’_main’]" -s "EXTRA_EXPORTED_RUNTIME_METHODS=[’cwrap ’]" -s ALLOW_MEMORY_GROWTH=1 Listing 2: WebAssembly compilation command for the array application

# Get pepper directory for toolchain and includes. # # If NACL_SDK_ROOT is not set, then assume it can be found three directories up. # THIS_MAKEFILE := $(abspath $(lastword $(MAKEFILE_LIST))) NACL_SDK_ROOT ?= $(abspath $(dir $(THIS_MAKEFILE))../..)

# Project Build flags WARNINGS := -Wno-long-long -Wall -Wswitch-enum -pedantic -Werror CXXFLAGS := -pthread -std=gnu++98 $(WARNINGS)

# # Compute tool paths # GETOS := python $(NACL_SDK_ROOT)/tools/getos.py OSHELPERS = python $(NACL_SDK_ROOT)/tools/oshelpers.py OSNAME := $(shell $(GETOS)) RM := $(OSHELPERS) rm

PNACL_TC_PATH := $(abspath $(NACL_SDK_ROOT)/toolchain/$(OSNAME)_pnacl) PNACL_CXX := $(PNACL_TC_PATH)/bin/pnacl-clang++ PNACL_FINALIZE := $(PNACL_TC_PATH)/bin/pnacl-finalize CXXFLAGS := -I$(NACL_SDK_ROOT)/include LDFLAGS := -L$(NACL_SDK_ROOT)/lib/pnacl/Release -lppapi_cpp -lppapi

# # Disable DOS PATH warning when using Cygwin based tools Windows # CYGWIN ?= nodosfilewarning export CYGWIN

# Declare the ALL target first, to make the ’all’ target the default build all: array.pexe clean:

A $(RM) array.pexe array.bc array.bc: array.cpp $(PNACL_CXX) -o $@ $< -O2 $(CXXFLAGS) $(LDFLAGS) array.pexe: array.bc $(PNACL_FINALIZE) -o $@ $<

# # Makefile target to run the SDK’s simple HTTP server and serve this example. # HTTPD_PY := python $(NACL_SDK_ROOT)/tools/httpd.py

.PHONY: serve serve: all $(HTTPD_PY) -C $(CURDIR) Listing 3: Makeﬁle of the array PNaCl application cd naclsdk update Listing 4: Installing the NaCl SDK

@ %* Listing 5: Make.bat ﬁle content of a PNaCl application set NACL_SDK_ROOT= set TOOLCHAIN=pnacl make Listing 6: PNaCl compilation commands cd bootstrap-vcpkg.bat vcpkg.exe integrate install vcpkg.exe install curl Listing 7: Installing curl for Visual Studio 2017 B Appendix 2

GitHub repository containing all relevant ﬁles.

B C Appendix 3

Figure C.1: Execution times of the Fibonacci application showing the execution time per browser.

Figure C.2: Load times of the Fibonacci application showing the load time per browser.

C Figure C.3: Execution times of the Array application showing the execution time per browser.

Figure C.4: Load times of the Array application showing the load time per browser.

D Figure C.5: Execution times of the Numeric application showing the execution time per browser.

Figure C.6: Load times of the Numeric application showing the load time per browser.