Evaluating Blazor

WebAssembly for the Progressive

Web Application

Front-End

A Comparative Study Using ReactJS as a Baseline

Main Subject area: Computer Engineering Authors: Vida Rashidi, William Segelström Supervisor: Denis Golubovic JÖNKÖPING 2021 August

This final thesis has been carried out at the School of Engineering at Jönköping University within Computer Engineering. The authors are responsible for the presented opinions, conclusions, and results.

Examiner: Johannes Schmidt Supervisor: Denis Golubovic Scope: 15 hp Date: 2021-08-19

i

Abstract

This study is conducted to evaluate the Blazor WebAssembly framework for the Progressive Web Application (PWA) methodology. A comparative study is conducted with a ReactJS PWA as a baseline. The two frameworks are evaluated in their front-end performance and documentation of PWA- focused subjects. Front-end performance is measured between two experimental applications that test the loading times, heap memory usage and loading consistency during layout generation. It is found that a Blazor WebAssembly PWA takes on average a range of 0.34, 0.18, and 0.06 seconds less time to generate a layout than a ReactJS PWA. The Blazor WebAssembly PWA was less consistent in its loading times when handling many elements. Documentation that covers Progressive Web Application terminology was found to be different between the frameworks. The Blazor WebAssembly documentation covers more topics and provides first-hand knowledge while the ReactJS documentation covers fewer topics and relies on external sources to provide the necessary explanations. These findings indicate that Blazor WebAssembly is a faster framework when updating large amounts of elements in comparison to ReactJS. However, the ReactJS PWA was found to be overall more consistent in its loading times. Documentation varied between the two frameworks. Documentation in Blazor WebAssembly covers more Progressive Web Application subjects and is more in-depth than ReactJS. This study only evaluates applications developed in ReactJS and Blazor WebAssembly. Both are tested on Google Chrome in a desktop environment. Keywords: Blazor WebAssembly, .NET, WebAssembly, ReactJS, JavaScript, Progressive Web Application, Virtual DOM

ii

Table of content Abstract ii 1 Introduction 1 1.1 Background 1 1.2 Problem statement 2 1.2.1 PWA and Web Frameworks 3 1.2.2 Layout Generation in a PWA 3 1.2.3 Why ReactJS Is Selected For Comparison 4 1.3 Purpose and research questions 4 1.3.1 Presentation of Research Questions 5 1.4 Scope and limitations 5 1.5 Disposition 6 2 Method and Implementation 7 2.1 RQ 1 - Literature Study 7 2.2 RQ 2 - Experiment Design and Execution 8 2.2.1 Implementation details 12 2.2.2 Specifications 16 2.2.3 Measuring Performance in Chrome DevTools 16 2.3 Method Motivation 17 2.4 Data collection 17 2.4.1 Gathering Performance Profiles for the Experiment 17 2.4.2 How the Experiment Is Automated 18 2.4.3 Collecting Information for the Literature Study 19 2.5 Data analysis 20 2.5.1 Experiment - Comparing Means of Independent Samples 20 2.5.2 Literature Study 22 2.6 Validity and reliability 22 2.6.1 Validity 22 2.6.2 Reliability 22 2.7 Considerations 23 2.7.1 Selection of Sources 24

iii

2.7.2 Implementation Fairness 24 3 Theoretical framework 27 3.1 Research Questions Related to the State of the Art 27 3.2 Technical Components of the Study 29 3.2.1 Progressive Web Application 29 3.2.2 WebAssembly 31 3.2.3 Virtual DOM 31 3.2.4 Blazor WebAssembly 32 3.2.4.1 C# Timer class 34 3.2.5 ReactJS 35 3.2.6 V8 JavaScript Engine 35 4 Results 38 4.1 PWA support for each framework 38 4.1.1 Blazor WebAssembly 39 4.1.2 ReactJS 41 4.2 Experiment Sessions 42 4.2.1 Session 1 - 720 pairs 42 4.2.2 Session 2 - 360 pairs 43 4.2.3 Session 3 - 180 pairs 44 4.3 Data Analysis 45 4.3.1 RQ 1 - Literature analysis 45 4.3.1.1 Analysis of results for Blazor Wasm 45 4.3.1.2 Analysis of results for ReactJS 45 4.3.2 RQ 2 - Experiment analysis 46 4.3.2.1 RQ 2.1 - Time difference 46 4.3.2.2 RQ 2.2 - Tree sizes 47 4.3.2.3 RQ 2.3 - Consistency in Loading Times 49 5 Discussion 51 5.1 Results discussion 51 5.1.1 PWA Support in the Blazor Wasm documentation 51 5.1.2 Virtual DOM differentiation 52

iv

5.2 Method discussion 54 5.2.1 When Analysing Memory From the Experiment 55 6 Conclusions and further research 57 6.1 Conclusions 57 6.1.1 Blazor WebAssembly as a PWA framework 57 6.1.2 Blazor WebAssembly and VDOM Generation 57 6.1.3 Practical implications 58 6.1.4 Scientific implications 58 6.2 Further research 58 7 References 59 8 Appendixes 64 Appendix 1 GitHub Repository of Source code for experiment PWAs and Python scripts used to collect and analyse retrieved data 64

v

1 Introduction

In this section, the reader is introduced to the elements that are included in this study as well as the problem statement. This section also includes background information required for understanding the purpose of this study. Cross-platform development is the development of software that is not dependent on a specific platform. There are multiple methodologies for developing cross-platform software. One methodology proposed for cross-platform development is Progressive Web Applications (PWA). PWAs are web applications, which means that any web framework can be used to develop a PWA. The large selection of frameworks is making it difficult to select the most appropriate one for PWA development. This is because no established web framework is presented as a PWA-specific one. For this study, selected frameworks are scrutinized in the aspects of support for the PWA methodology and their front-end performance. The scrutiny is done by a comparative study between the two web frameworks. The two selected frameworks for this study are Blazor WebAssembly (Blazor Wasm) and ReactJS. The goal of this study is to create an understanding of how the selected frameworks support the PWA methodology and how well their front-end performs in a PWA runtime environment.

1.1 Background PWAs are web applications that can be installed locally. PWAs are installed by visiting a website that fulfils the PWA requirements and via the browser interface downloads the application. Figure 1 shows how the installation prompt appears to the user. Google Chrome is a web browser that is part of the Chromium project, which is an open-source initiative by Google. Amongst the requirements for a valid PWA is the inclusion of a service worker, this is a separate script that must listen for a set of events.

Figure 1: Installation prompt on Google Chrome User engagement is why developers need to use a fast and efficient front-end framework to develop their application. If an application takes a long time to load it increases the probability of a bounce. A bounce is defined as a user arriving at a page but then leaves during loading (An, 2018). The front-end of a framework handles the look and behaviour of the user interface, which in turn determines the user experience.

1

Both Blazor Wasm and ReactJS are front-end frameworks, making user engagement relevant for this paper. WebAssembly and JavaScript are standards in programming for web browsers. Blazor Wasm executes its source code by using a runtime written in WebAssembly. The Blazor Wasm runtime is interpreted by the browser. ReactJS is written in JavaScript and its source code is directly interpreted. Both Blazor Wasm and ReactJS provide a programming interface for the developer to alter the user interface. The interfaces achieve this by interacting with the Document Object Model. The Document Object Model (DOM) is a programming interface used to alter Hypertext Markup Language (HTML) documents as nodes in a tree structure. HTML documents along with additional styling represent the user interface on all web applications. The goal of DOM-altering interfaces found in Blazor Wasm and ReactJS is to streamline user interface development by providing generalised code syntax and auto-generation of DOM instructions. Auto-generation of DOM instructions are done using the Virtual DOM (VDOM) technique. When using VDOM, a light copy of the entire DOM is saved in memory to compare with the real DOM. Both frameworks use their own implementation of this technique. With WebAssembly being used for generating layouts for the DOM it is interesting to investigate what performance difference there is in comparison to established JavaScript frameworks.

1.2 Problem statement In this section, the problem is outlined with a smaller background supported by the state of the art. The presented problems regard the lack of PWA-centric frameworks and undocumented front-end performance of the Blazor WebAssembly framework. Developers at Google are developing additions to the Chromium project to further support PWAs running on the browser (Chrome Team, 2021) and argues that applications that have been re-developed to a PWA have a decreased bounce-rate (Google Chrome Developers, 2018). This is due to faster loading times and smaller file sizes when compared to the application before re-development. PWAs occupies a small portion of all web applications on the web. However, there is a steady increase in registered service workers (HM, 2020) indicating growth in PWAs. Hence, there is an indication of increasing adoption and support for PWAs. With the increasing adoption and support of PWAs, there are few indicators of which frameworks are best suited for PWA development. Along with Blazor Wasm being a new release, it is not documented how well it performs on the front-end in a PWA environment.

2

1.2.1 PWA and Web Frameworks Biørn-Hansen et al. (2018b) propose that more web frameworks should be evaluated for PWA- and cross-platform development. For this study selected frameworks are only evaluated for PWA development. In the case of other methodologies, there are technical frameworks created with the methodology in mind. Biørn-Hansen et al. (2018a) conducted a taxonomy and survey of cross-platform frameworks where they connected the hybrid-, interpreted and model-driven approach to other frameworks. Unfortunately, they could not relate the PWA methodology to any framework.

1.2.2 Layout Generation in a PWA Blazor Wasm utilizes a runtime written in WebAssembly to execute its source code. The WebAssembly standard has been evaluated by other papers regarding execution times. However, the WebAssembly applications they have compiled from have been written in a low-level language. In a paper by Haas, et al. (2017) they conducted a benchmark on WebAssembly and JavaScript applications. It was found that web applications written in WebAssembly have faster execution times on mathematical equations in comparison to those written in JavaScript. Cross-compiling JavaScript to WebAssembly has shown speed improvements in executing heavy calculations in the Reiser and Bläser paper (2017). One caveat with their method is that they rely on the developer to write safe code that JavaScript interpreters otherwise protect from. WebAssembly has an architectural difference where it is interpreted before running the application. JavaScript is interpreted during the execution of the application and is presented as a bottleneck in Price and Eberhardt (2018) white paper regarding WebAssembly. WebAssembly applications have faster execution when compiled from lower-level languages. It remains untested how fast a WebAssembly application is when compiled from a high-level coding language. This is what Haas, et al. (2017) suggests for future research in their paper. WebAssembly has been measured to garner a faster web application when compared to JavaScript, but the standard has not been measured in how it can change VDOM generation times. While the previously mentioned studies focused on execution speed, little has been said in academia about the memory allocation differences. Krause (2018) created an interactive benchmark where he evaluated a collection of JavaScript and WebAssembly front-end frameworks in memory allocation. However, Blazor Wasm has yet to be evaluated.

3

1.2.3 Why ReactJS Is Selected For Comparison Today web developers use JavaScript frameworks including Angular, ReactJS, and JQuery (stylised jQuery) to develop most web applications where jQuery is among the most used web frameworks. But there is an ongoing trend shift in popularity where jQuery is losing popularity to ReactJS and Angular (Stack Overflow, 2020). ReactJS is used for PWA development in two previous studies. One being in the study conducted by Biørn-Hansen et al. (2018b) when comparing PWA against applications developed using other methodologies. The second being the application in Huynh and Nguyen-Ky (2020) case study. The DOM interface of ReactJS is measured as the fastest among other existing JavaScript frameworks when altering the DOM using a framework (Persson, 2020). Its popularity, usage in academia researching PWA and its performance in VDOM generation are considered when being selected as a comparative baseline.

1.3 Purpose and research questions With PWA not having a specific framework entitled to it when compared to other methodologies, we can draw from the problem statement that an investigation is needed. Papers by Haas et al. (2017), Reiser, Bläser (2018), and Wang et al. (2019) compared perceived performance between WebAssembly and JavaScript. However, none have used a high-level language as source code for the WebAssembly implementation. Nor did the WebAssembly applications evaluated in the previous papers, use a runtime as Blazor Wasm does. ReactJS has previously been benchmarked in both its performance and DOM-capabilities by Krause (2018) and Persson (2020), but not Blazor WebAssembly. This indicates that there is little known about Blazor Wasm’s performance when generating layouts and what difference a WebAssembly framework can provide in that aspect. Summing up the two problem statements, a purpose is formed: To evaluate the differences that the Blazor Wasm framework can yield for PWA front- end development compared to the ReactJS framework. Using the results obtained from this research we can conclude how a WebAssembly framework impacts the PWA front-end. We can also provide general indicators for which framework, between ReactJS and Blazor Wasm, to select when developing PWAs. This indication is limited to provide documentation and performance when handling changes in the user interface, further limitations are provided in 1.4 Scope and limitations.

4

1.3.1 Presentation of Research Questions To measure how the Blazor Wasm framework supports PWA development entails the following question: [1] To what degree does Blazor Wasm support PWA development in comparison to ReactJS? For a PWA to be reliable according to Richard and LePage (2020) it needs to be performant. To understand the differences in front-end performance the following questions arises: [2] How does a Blazor Wasm application differentiate itself from established JavaScript frameworks in VDOM generation performance? [2.1] How much time does it take to generate a VDOM tree between the two frameworks? [2.2] What are the size differences of VDOM trees in memory between Blazor Wasm and ReactJS? [2.3] How consistent is each VDOM implementation in generation times?

1.4 Scope and limitations Both research questions are limited to the frameworks Blazor Wasm and its baseline, ReactJS. Additionally, the study is conducted within the research area of PWA development. As RQ 1 states, this paper investigates the amount of PWA support provided in the Blazor Wasm framework. Support is defined by how thoroughly PWA subjects are documented. Through a literature study, Blazor Wasm and ReactJS support of PWA development is evaluated. This is done by using the official documentation and YouTube channels for respective frameworks. In this study, PWA support is defined by and limited to the following factors: ● Documentation covering PWA development in relation to the framework ● PWA specific features embedded in the framework ● External PWA tools provided by the framework organisation Programming language for each framework is not put under scrutinization in this study, this is to focus on the implementation of each framework. For RQ 2, the different approaches for VDOM generation are evaluated. This is achieved through experimentation on two PWAs, one created with Blazor Wasm and the other with ReactJS.

5

With speed being part of the reliability factor in a PWA (Richard & LePage, 2020), two factors regarding loading times are regarded when comparing VDOM implementations. Memory usage is investigated to scrutinise other aspects of resource efficiency. For the second research question, differentiation of VDOM is defined by three parameters: ● How fast the framework can generate a VDOM tree ● Space in memory used to store the VDOM tree ● Loading time consistency The lack of time for testing and experimenting on a wider scope is a limitation of this study. Aspects covered by the wider scope include testing the experiment on multiple web browsers and using other popular front-end frameworks like jQuery or Angular.

1.5 Disposition Chapter 1 Introduction, this chapter introduces the research area and presents the gap that is being researched. The research questions are presented, and their selection is motivated. Further, the scope of the problem area and its limitations are also defined. Chapter 2 Method and Implementation, this chapter explains the chosen method for experimentation and the motivation for its selection. This chapter also presents the chosen method for the collection and analysis of the data while also evaluating its validity and reliability. Chapter 3 Theoretical Framework, this chapter presents the state-of-the-art theories and methodologies that are scrutinized. Also, further theories and sources are presented to aid in the understanding of the research questions. This chapter allows the reader to get a deeper understanding of the elements in the research. Chapter 4 Results, the obtained results from the experiment are presented and analysed in this chapter. The results’ relevance to the research questions is presented and discussed. Further, conclusions are drawn based on the obtained results which answer the research questions. Chapter 5 Discussion, in this chapter the choice of method and obtained results are discussed. The chapter presents and discusses the limitations of the study and suggestions for improvements. Chapter 6 Conclusions and further research, this chapter gives proposals for prospects in research work and presents the conclusions that have been drawn from the study.

6

2 Method and Implementation

This section describes two methods used in this study. Each method description is divided into sub-chapters based on the research question it aims to answer with the prefix RQ [Number]. The research questions are answered by conducting a combination of a literature study and an empirical study. By conducting a literature study, data and information is collected from various sources and analysed to provide the answer for RQ 1. The empirical study includes an experiment that is studied along with the results it generates. As part of the experiment, two PWAs are created, where one is implemented using Blazor Wasm and the other using ReactJS. The obtained results provide the answers for RQ 2 and its sub-questions.

2.1 RQ 1 - Literature Study This literature study aims to answer research question 1 (RQ [1]) with a qualitative approach. [1] To what degree does Blazor Wasm support PWA development in comparison to ReactJS? This theoretical literature study is focused on finding sources covering PWA subjects and text regarding the three aspects of PWA support. The PWA subjects are defined by aspects presented in the framework that are coupled with the PWA methodology. The keywords used are outlined in 2.4.3 Collecting Information for the Literature Study. A set of sources are used to get relevant literature covering each framework. The sources are maintained by teams and community members working on the framework. Also, each source is either hosted or endorsed as official by each respective framework organisation. The sources for investigating Blazor Wasm are: ● Official Blazor documentation ● The dotNet YouTube channel And the sources for investigating ReactJS are: ● ReactJS Documentation on Create React App ● Facebook Developers YouTube channel Collected information is categorised on the PWA subject it covers. With this information, a summary is created for each framework where a comparison is made regarding the extent of documentation made on selected topics.

7

2.2 RQ 2 - Experiment Design and Execution This method aims to answer research question 2 (RQ [2]) and its subset of questions: [2] How does a Blazor Wasm application differentiate itself from established JavaScript frameworks in VDOM generation performance? [2.1] How much time does it take to generate a VDOM tree between the two frameworks? [2.2] What are the size differences of VDOM trees in memory between Blazor Wasm and ReactJS? [2.3] How consistent is each VDOM implementation in generation times? The method is an experimental simulation as defined by Felderer and Guilherme Horta (2020) where it is conducted in a controlled environment with some realistic factors involved in the experiment environment (p. 41). Two PWAs are developed in Blazor Wasm and ReactJS for the experiment. Both are built following the installable requirements set by Chrome. The main activity of this experiment is to trigger layout generation from each front- end framework. By altering the DOM, the framework is forced to generate a new, or alter the current VDOM tree and apply those changes through the instructions provided by its differentiation algorithm. Each PWA initially creates a set of nodes in a tree structure. Every node contains a numerical value that is altered continuously. Every node is removed when the instance is finished. A flowchart visualising the order and what actions the experiment application executes is seen in Figure 2.

8

Figure 2: Flowchart of the experiment application The experiment is split up into three sessions. Each session has a different number of initial node pairs (푛푝푎푖푟푠). See Table 1 for the distribution of node pairs for each session. Notice that the max amount is only lowered from the highest value of 720 pairs. Limitations in the number of elements and the node structure are kept under the Lighthouse DOM size limitations as defined in Web.dev (2019). This is to retain realism in the experiment.

9

Session name npairs

Session 1 720

Session 2 360

Session 3 180

Table 1: Experiment sessions and the amount of max elements The activities conducted in the experiment applications are inspired by Persson’s (2020) experiment on evaluating ReactJS VDOM and Procida’s ReactJS Fiber demo (2017). The experiment application executes the following tests in the given order:

1. 푛푝푎푖푟푠 elements of type

are created. Each

have a

element

resulting in 2푛푝푎푖푟푠 elements in total. The elements are distributed by the following rules: ○ Each

and

pair is distributed so that there is a maximum

number of pairs (푛푝푎푖푟푠). ○ The element tree is 32 elements deep. ○ Every node that is not on the 32nd level of depth has between 1 - 60 child nodes. 2. For 30 seconds, data within all

elements are changed within a repeating time interval of one second. 3. All elements created in step 1 are de-referenced. An overview of how the element tree is structured is viewed in Figure 3. In the figure, coloured nodes are elements while uncoloured nodes are inner data that are altered during the experiment.

10

Figure 3: DOM element structure of the tree The measurement starts when the application is fully loaded, and the start button is pressed on the interface. The application proceeds to alter the state of the page at a one-second interval without input from the user. The Blazor application and the ReactJS application are measured in separate sessions. When changing data, the application invokes the JS / CSS > Style > Layout > Paint > Composite cycle according to the pixel pipeline. Read more about the pixel pipeline in 2.2.3 Measuring Performance in Chrome DevTools. Independent variables of the experiment are: ● Element amount ● Testing time seconds ● Maximum tree depth ● Maximum child amount per node Dependent variables are modified by the independent variables: ● Partition of total time spent on loading ● Tree size in memory ● VDOM generation times

11

Intersecting variables between the independent and dependent variables are for the ● Blazor Wasm PWA ○ Blazor Wasm implementation ○ .NET ○ Mono WebAssembly runtime ● ReactJS PWA ○ ReactJS implementation ● Both implementations ○ JavaScript engine interpreting front-end code Gathered data after an analysis is used to answer RQ 2, RQ 2.1, RQ 2.2 and RQ 2.3.

2.2.1 Implementation details A recursive method is used to generate the DOM tree where an add node function is executed to create a root node. The add function calls itself to add child nodes to the parent node and thereby creates a recursive chain of calls. The independent variables are represented by parameter constants in the source code. To change the values of these variables a redeployment of the PWAs is required. Parameter names are the same between the two projects and are mapped to the independent variables as shown in Table 2.

Parameter constant name Represents independent variable

PARAMETER_CONTENT_ELEMENT_AMOUNT Pair amount (npairs)

PARAMETER_MAXIMUM_TREE_DEPTH_AMOUNT Maximum tree depth

PARAMETER_MAXIMUM_CHILD_NODE_AMOUNT Maximum child amount per node

PARAMETER_EXPERIMENT_TIME_SECONDS Testing time seconds

Table 2: How constant variables in the source code are mapped to independent variables The Blazor Wasm and ReactJS source code can be seen in Figure 4 and Figure 5. DistributableElementsAmount is a global variable that determines how many element pairs can be created and its initial value is set to the parameter constant representing element amount.

12

NodeInstance AddElement(int data, int treeDepth) { NodeInstance element = new NodeInstance(data); DistributableElementsAmount -= 1; if (treeDepth > 0) { for (int i = 0; i < PARAMETER_MAXIMUM_CHILD_NODE_AMOUNT; i++) { if (DistributableElementsAmount > 0) { element.ChildNodes.Add(AddElement(data + 1, treeDepth - 1)); } else { break; } } } return element; }

Figure 4: AddElement method in Blazor Wasm

13

function addElement(data, treeDepth) { let elementToBeAdded = new NodeInstance( data, DistributableElementsAmount ) DistributableElementsAmount--

if(treeDepth > 0) { for(let i = 0; i < Parameters.PARAMETER_MAXIMUM_CHILD_NODE_AMOUNT; i++) { if(DistributableElementsAmount > 0) { elementToBeAdded.addChild( addElement(data + 1, treeDepth - 1) ) } else { break } } } return elementToBeAdded.toObject() }

Figure 5: addElement method in ReactJS To modify data on the tree, another recursive method is used. The data updating function called ScrambleTree takes a node as an argument, changes its data by incrementing the data by one, and then invokes its own function to modify its child node or nodes. Figure 6 and Figure 7 presents how the ScrambleTree method is implemented in each framework.

14

void ScrambleTree(NodeInstance rootNode) { rootNode.AddToData(1); if(rootNode.ChildNodes.Count <= 0) { return; } foreach(NodeInstance childElement in rootNode.ChildNodes) { ScrambleTree(childElement); } }

Figure 6: ScrambleTree method in Blazor Wasm

function scrambleTree(rootNode) { rootNode.data += 1 if(rootNode.childNodes.length <= 0) { return rootNode; }

for(let childElement of rootNode.childNodes) { childElement = scrambleTree(childElement); }

return rootNode }

Figure 7: scrambleTree method in ReactJS Timing is handled by two different implementations in each application. For the Blazor Wasm application, an instance of the standard .NET class called Timer from the System.Timers namespace is used. The Timer class invokes an overridable event method when its interval reaches zero. For the ReactJS application, the Window.setInterval (or just setInterval) function is used. The setInterval function is a defined web standard available for web development. In the same vein as the Timer class the setInterval fires a selected

15

function within a specified interval, for this experiment it is set to a one-second interval. The main difference between the functions is the return statement. In the ReactJS implementation, the method returns an instance to the parent node when reaching the bottom of the tree. While the Blazor Wasm implementation does not return an instance. This is because it was found during the study that when calling methods with a class parameter within the interval invocation method, it takes the class instance as a reference and thereby changes the value using a reference instead of returning an instance.

2.2.2 Specifications The experiment is conducted on a personal computer. The specifications of this computer are: ● Operating System - Windows 10 Education Edition 64 bit ● CPU - AMD A10 PRO-7350B R6, 10 Compute Cores 4C + 6G (4 CPUs), ~2,1GHz ● RAM - 8 GB (6,95 GB Usable) The ReactJS project is dependent on the following React-centric scripts: ● React – Compatible with version 17.0.2 ● React-dom – Compatible with version 17.0.2 ● React-scripts – Version 4.0.3 The Blazor project is compiled using .NET runtime version 4.0.50524.0 in interpreted mode, the experiment is run as a PWA on Chrome Version 90.0.4430.93 and both PWAs are served via the Azure platform as static web pages.

2.2.3 Measuring Performance in Chrome DevTools To measure and evaluate tree generation time, Chrome DevTools is used. The tool analyses performance and accessibility for web applications. The measurement tool is placed in the “Performance” section of the DevTools tab. This tool measures according to the pixel pipeline. The pixel pipeline is a chronological order of steps executed to load a page. It is divided into three portions. These portions are JavaScript and two rendering portions where the JavaScript portion is first (Lewis, n.tag">d.). The order can be seen in Figure 8. In this study, the JavaScript portion is analysed.

16

Figure 8: Lewis, P. (2019). JS / CSS > Style > Layout > Paint > Composite [Figure]. Retrieved 20 April 2021, from https://developers.google.com/web/fundamentals/performance/rendering/ It is during the JavaScript portion of the pixel pipeline in which Blazor and ReactJS generate their layouts. By measuring the microseconds to execute the JavaScript portion (Jduration) it is possible to gather the time it takes to generate a VDOM Tree. Then by gathering the difference in memory usage between the two points of starting (Jstart) and finishing (Jstart + Jduration) the VDOM-tree size is gathered. The event that invokes the JavaScript portion is named TimerFire and is searched for when analysing the profiling files. TimerFire is a complete event and is categorised as a phase-type by the letter X.

2.3 Method Motivation The chosen method consists of a combination of a literature study and an empirical study. This gives a qualitative and quantitative approach to the purpose of the study. RQ 1 is approached by analysing the official documentation of both frameworks as well as videos found on official YouTube channels. The purpose of this is to understand the relation of the PWA with the frameworks and how well each framework supports the PWA methodology. Studies indicate that 18-22 % of developers, regardless of experience, prefer watching videos as a source for learning new skills (DevelopIntelligence, 2018). Hence, the official YouTube channels belonging to the two companies have been selected as sources for information collection. In the empirical study, the tests are designed to understand the page loading times for each application. This provides results that indicate whether there is a difference in page loading times which answers RQ 2.

2.4 Data collection The data gathered from the literature study is selected depending on its relevance to the research area and more specifically how well it provides the answer for RQ 1.

2.4.1 Gathering Performance Profiles for the Experiment Google DevTools for Chrome is used to record performance variables. Google DevTools is fitting for the measurement since it is integrated within the test

17

environment and provides the needed variables. The toolset records according to the pixel pipeline, meaning it separates rendering tasks from invoked JavaScript methods. Data is collected by creating a snapshot recording called a profile. The creation is initialised from the performance tab of DevTools. For this experiment the main thread is recorded, this is because Blazor’s and ReactJS’ runtime is run on the main thread only. When the website layout changes, the following steps according to the pixel pipeline is executed: JS > Style > Layout > Paint > Composite The measurement only collects data for the JavaScript portion. From the gathered profile a set of data is selected. The selected variables are: ● Time to generate layout ● Tree size in memory ● The portion of total time used to generate layout Time until re-rendering is the time used to execute the JavaScript portion of the pixel pipeline, this is selected to measure the time of generating a VDOM tree. This is saved in microseconds. Data regarding memory usage is collected due to it representing how much space a VDOM tree takes in memory. The VDOM tree size is collected as bytes. Using a selective approach, the data points are picked. Data is only collected when a layout change is invoked. This is done to avoid collecting irrelevant data. A sample of 30 measurements from each implementation is collected. Each sample has its own set of data points. The portion of total time used to generate a layout is the fraction of the total time used to generate one VDOM tree. This sample's standard deviation is used to measure the consistency of each VDOM generation technique. The fraction is a value between zero and one. Every profile is saved as a JavaScript Object Notation (JSON) file for a separate analysis session. The JSON file type is a structured type that uses keys and associated collections to identify data. It is also widely accepted and can easily be read by other applications.

2.4.2 How the Experiment Is Automated The experiment is conducted using an automatization script. This script is written specifically for this study. It is a Python 3.8.2 script for 32-bit systems. The script takes control of the computer interface by managing the computer mouse and keyboard to forward the experiment.

18

The library PyAutoGUI (PAG) is used for mouse, keyboard, and user-interface control. Along with PAG, the script uses standard libraries included in the current Python version which is the Time- and OS library. OS is used for starting the PWA and Time is used to get timestamps when the experiment is done. Before starting the script, it must be provided with the following values and resources: ● App-id – Unique identification string provided for specific PWA ● Window title – Title of the application shell window on the interface ● Reference image of the start button – A image of the start button seen on the user interface ● Button image name – Name of the reference image file used The app-id parameter is a unique identification string for an installed PWA. The app- id string is retrieved by inspecting the desktop shortcut created by Chrome when installing the PWA. The window title is a string that represents the title of a process window on the operating system. A reference image and the name of the reference image need to be supplemented for the script to find the start button in the application interface. The reference image is a screen capture of the button element when the PWA window is maximised. When the script has reached its done state, it saves the profile and finishes the instance.

2.4.3 Collecting Information for the Literature Study The collection of information is done through a literature study involving official sources that belong to the two companies. Articles and videos from each source are searched for using the search functionalities found within each website. The terms used for searching articles and videos are “PWA” and “Progressive Web Application”.

19

Keywords are used in searching through selected sources. The keywords are related to features that are specific to PWAs. The keywords are based on conceptual terms and features found during the initial literature search. These being terms from Altexsoft (2018), terms from Mozilla (2021), and features showcased in Whatpwacando (Moerkerke, n.d.). The selected keywords are: ● Offline mode ○ Caching pattern ○ IndexDB ○ LocalStorage ● Push notifications ● Service worker ● Manifest ● Hardware access terms ○ Camera ○ Microphone ○ Geolocation ○ Vibration (Mobile) ○ Device orientation and motion On studying the documentation of each framework, the differences are sought after and noted. The amount of PWA subjects covered in the documentation indicates the degree to which support is provided by the framework. In the literature study, it is also sought after the PWA specific features embedded in each of the frameworks. Furthermore, it is also studied if the framework companies provide external tools that aid in the development of PWAs. These observations along with the documentation content collected and presented in the upcoming chapter. Thereafter, the data can be analysed to draw conclusions and provide the answer for RQ 1.

2.5 Data analysis The authors explain the Z-test and standard deviation equations used on gathered data from the experiment. Also, how search words are used in the literature study and utilised in the selection of literature data.

2.5.1 Experiment - Comparing Means of Independent Samples There are two sets of samples, the Blazor Wasm sample from the Blazor Wasm population and the ReactJS sample from the ReactJS population. For every retrieved variable in each sample, a standard deviation and mean are calculated. Tree generation times, tree size and time spent on generating during layout

20

generation have their own mean and standard deviation calculated. Standard deviation is used to understand the consistency of the dependent variables. The equation for standard deviation is defined in Equation 1.

푛 ∑ 푥2 − (∑ 푥)2 푠 = √ 푛(푛 − 1)

Equation 1: Standard deviation

n = total amount of data x = data point value from s = resulting standard points sample deviation

The standard deviation is used to conduct a Z-test. A two-sample Z-test provides the differentiation between the populations (µR and µB) using mean values and standard deviation between the two samples. When performing the Z-test, the following hypotheses are tested:

Null hypothesis (H0): Blazor Wasm performs near equally as well as its ReactJS counterpart (µR - µB = 0).

Alternative two-tailed hypothesis (H1): There is a difference in

performance between the Blazor Wasm- and ReactJS PWA (µR - µB ≠ 0).

H0 and H1 is a question regarding equality between the samples from each population, hence making this a two-tailed test. The Z-value equation is defined in Equation 2. 푥̅̅̅̅̅̅̅̅̅ − 푥̅̅̅̅̅̅̅̅ 푧 = 푏푙푎푧표푟 푟푒푎푐푡 푠2 푠2 √ 푏푙푎푧표푟 + 푟푒푎푐푡 푛푏푙푎푧표푟 푛푟푒푎푐푡

Equation 2: Z-value

z = z-value 푥̅ = amount of data points in the sample

The confidence interval (α) is set to 0,05. Meaning that the experiment is analysed with 95% confidence that the results are correct. The critical value (p) based on the selected confidence interval is approximately 1,95996. If z > p or z < -p then the null hypothesis is rejected, and it is reasonable to suggest a difference in the tested variable between the two samples. If proven otherwise, there is no significant difference between the populations.

21

2.5.2 Literature Study PWA subjects are searched for within the content of the gathered data. Gathered data are categorised by its source and what subject it covers. If the selected data point does not contain any keyword from the list of subjects, then it is marked as missing. For each framework these three aspects are summarized: ● Documentation covering PWA in relation to the framework ● PWA specific features embedded in the framework ● External PWA tools provided by the framework organisation

2.6 Validity and reliability Various cautions have been taken during the empirical study as well as the literature study to increase the validity and reliability of the research. Examples of these precautions are multiple iterations when conducting the empirical study and using reliable and credible sources for the literature review.

2.6.1 Validity The memory difference gathered during the experiment is used to analyse the size of the generated DOM tree, this is valid because VDOM trees are stored in memory. The two sets of values for memory usage and page loading are utilised to formulate an answer for RQ 2. By measuring the time and resources it takes to execute the JavaScript portion upon altering the layout data for each framework RQ 2 and its sub-questions can be answered. RQ 2 is a descriptive question where it mainly requires quantitative data to be fully answered, therefore a dataset with multiple data points containing layout generation time and heap memory changes suffice. The literature search is done by gathering data for RQ 1. With qualitative knowledge of the source, it is possible to conclude with a descriptive analysis of how each framework supports PWA development in the selected factors detailed in 1.4 Scope and limitations.

2.6.2 Reliability Google DevTools has its source code open for sharing and reworking with its source code, as of writing, is updated frequently. It is developed by the Google Chrome team and is fully integrated with the Chrome web browser. The collection of data is done by reading performance measurement files provided by the development tools in Chrome. A Python script is used to automate the data collection process.

22

Data files are read by another Python script that picks samples by category and function name. With collected data, the script calculates z-value, critical value, and standard deviation using the two samples. The statistics module of the SciPy library is used to gather the critical value according to the z-table. The reason for automating the collection and calculation of data is to decrease human error and time consumption. Moreover, the experiment is designed to prevent any of the frameworks from having an advantage over the other. Both applications perform the same tasks and are tested using the same utilities, such as the same computer and the same version of Chrome. Also, background applications are stopped to limit their impact on the experiment results. To keep likeness in both implementations, the code is written using built-in methods and primitives with the same functionality if there is no other preferred way in the respective language. By the same function meaning characteristics and implementations that are alike. Structure and the variable names of the source code are written alike. See attached code in 2.2.1 Implementation details and 8 Appendixes for practical examples. Blazor Wasm is only compared with ReactJS for each research question and method in this study. This is done to keep results consistent. To further increase the reliability of the experimentation and allow for future research, the PWAs and automatization scripts are published on a public GitHub repository. The experiment is conducted 30 times per implementation to ensure that the obtained population is not impacted by interfering environmental factors, like available computer resources, automatic background tasks etc. This is done along with the analysis to gain a plausible statistical significance. Furthermore, the literature study is based on reliable sources of information. To gather information and conduct the literature study, sources are critically reviewed. The selected sources are close to each framework and include minimal irrelevant information regarding each framework.

2.7 Considerations In this section, the considerations of this study are outlined. Considerations regard the selection of information resources for both literature studies conducted in this study. Moreover, how source code is written in each experimental application to ensure a fair measurement is outlined.

23

2.7.1 Selection of Sources Numerous different sources are used to conduct the preliminary literature study to understand the research area. Hence, critically reviewing these sources and their credibility is an important consideration. To ensure that sources for the literature study method are reliable, only official sources belonging to the respective company have been used for gathering information. Some of these sources refer to other external sources, also these are critically reviewed during the study. Since videos are a part of the literature study, the content makers, as well as the contributors in the videos, are assessed to ensure their knowledge and background. Moreover, throughout the research, referencing correctly, fairly, and ethically has been taken into consideration.

2.7.2 Implementation Fairness The scientific aspects that have been taken into consideration are to design the experiment to be fair to both frameworks. The source code of the two PWAs is written to minimize the risk of giving one advantage over the other. The experiment is conducted on the same computer and uses the same version of Chrome. During the experimentation, it is ensured to stop non-vital applications running in the background that can interfere with the results of the experiment. Code style in both projects is selected to fit each programming language which considers the placement of curly brackets and indentation styles. Therefore, the placement of the curly brackets is different between the NodeInstance class declarations as seen in Figure 9 and Figure 10.

24

class NodeInstance { constructor(data, id) { this.data = Number(data) this.childNodes = Array(0) this.id = Number(id) } toObject() { return { data : this.data, childNodes : this.childNodes, id : this.id } } addChild(element) { this.childNodes.push(element) } addToData(addition) { this.data += Number(addition) } } export default NodeInstance

Figure 9: Class declaration in NodeInstance.js

25

using System; using System.Collections.Generic; namespace pwa_blazor.Models { public class NodeInstance { public int Data { get; set; } public List ChildNodes { get; set; }

public NodeInstance(int data) { Data = data; ChildNodes = new List(); } public void AddChild(int data) { ChildNodes.Add(new NodeInstance(data)); } public void AddToData(int amount) { Data += amount; } } }

Figure 10: Class declaration in NodeInstance.cs

26

3 Theoretical framework

This section is populated with detailed background information and theory about the different components included in this study. The research questions are related to the state of the art found within each respective area. The structure of this section is a research question followed by the technical components of the state of the art. Blazor Wasm and ReactJS are frameworks that can be utilized for the creation of PWAs, thus certain aspects of the frameworks are chosen to be evaluated in this research.

3.1 Research Questions Related to the State of the Art RQ [1] To what degree does Blazor Wasm support PWA development in comparison to ReactJS? Blazor Wasm had its first preview release in 2019 as a part of ASP .NET Core 3.0 and its first stable release on the 19th of May 2020 (Roth, 2020a). This indicates that Blazor Wasm is a relatively new framework. Included with Blazor Wasm’s first preview release was a PWA template project, entailing its intended use as a framework for PWA development. ReactJS is a well-established web framework where its first public version came out on the 2nd of July 2013 on their GitHub repository (O'Shannessy, 2013). ReactJS has been used to create a PWA in Biørn-Hansen et al. (2018b) evaluation of cross- platform methodologies. Huynh and Nguyen-Ky (2020) conducted a case study on a PWA developed in ReactJS. The study focused on web-based applications for facility management. In this study, they claimed that ReactJS was a good choice for PWA development. Like Biørn-Hansen et al. (2018b) they considered the PWA methodology as a platform unifier. Mena et al. (2019) implemented a PWA gathering geospatial data. In their preliminary literature study, they looked at PWA features supported in the Chrome browser on different platforms. It is not defined in their report which front-end framework they used for their PWA. On inspecting their deployed demo GeoNat (Mena et al., n.d.), it appears as they opted for AngularJS as a front-end framework for their PWA. They concluded that the development of a PWA made it easy to broaden the platform compatibility. But they did not detail why they selected AngularJS as the front-end framework for their PWA.

27

[2] How does a Blazor Wasm application differentiate itself from established JavaScript frameworks in VDOM generation performance? [2.1] How much time does it take to generate a VDOM tree between the two frameworks? [2.2] What are the size differences of VDOM trees in memory between Blazor Wasm and ReactJS? [2.3] How consistent is each VDOM implementation in generation times? WebAssembly has been measured as being consistent in performance. Reiser and Bläser (2017) garnered a decrease in execution time in their Speedy.js project. In their study, the label decrease in execution time as “speedup” and garnered ranges from 2% to 364% speedup in 14/16 cases. Regarding the remaining two cases, one had the same speed as its TypeScript source and the other suffered a -2% speedup metric. All sixteen cases consisted of eight TypeScript scripts compiled to WebAssembly running on two browsers each. Selected browsers for their empirical study were Chrome 59.0.3071.86 and Firefox 53.0. Haas et al. (2017) measured a 37% faster average when running C programs compiled to WebAssembly bytecode. This was in comparison with C programs compiled to asm.js code. The applications were part of the PolyBenchC suite, which is a collection of performance heavy calculations. Wang et al. (2019) had a similar method to Reiser & Bläser (2017) where they used WebAssembly as a compilation target for JavaScript. However, the goal of Wang et al. (2019) was to obfuscate the JavaScript source code by compiling the JavaScript to WebAssembly and obfuscating the generated byte code. On comparing obfuscated code by JSPro against other commercial tools they found a range of 15% - 32.8% reduction on executing time. In Krause's (2018) benchmark he measured memory usage in ReactJS 16.4.1, Yew and StdWeb. Yew and StdWeb are, like Blazor Wasm, front-end frameworks running on WebAssembly. For the benchmark, the applications conducted the following actions: ● Created 1000 rows in a table right after the page loaded (ready memory) ● Updated every tenth row five times (run memory) ● Replaced all rows five times ● Replaced and recreated all rows five times In Table 3 the memory usage (memory amount +- standard deviation) after a certain action is presented. Worth noting is that Yew is non-dependant on keys, this means that it can reuse existing DOM elements without having to instantiate new ones.

28

Action name Yew 0.5.0 StdWeb 0.4.8 React 16.4.1

Initial memory 2,5 +- 0,1 2,5 + − 0,2 2,8 + − 0,2

Create 1000 rows 4,7 +- 0,0 3,1 + − 0,0 6,7 + − 0,0 in a table

Update each 10th 5,4 +- 0,0 3,1 + − 0,0 7,6 + − 0,0 row for all rows five times

Replace all rows 5,3 +- 0,0 4,3 + − 0,2 7,9 + − 0,0 five times

Creating and then 14,9 +- 0,2 3,6 + − 0,1 3,8 + − 0,0 clearing all rows five times

Table 3: Memory usage +- standard deviation in Megabytes. Adapted from: Krause, S. (2018, September 27). Results for JS web frameworks benchmark – round 8. Stefan_Krause.blog(). Retrieved May 16, 2021, from https://stefankrause.net/js- frameworks-benchmark8/table.html

3.2 Technical Components of the Study In this section, the technical components and methodologies used in the study are explained. The explained items are PWA, WebAssembly, Virtual DOM, Blazor Wasm and ReactJS.

3.2.1 Progressive Web Application Progressive Web Application (PWA) is characterized as a type of software that is executed in a dedicated web browser using web- and native technologies. Browsers dedicated to one site are called a Site-specific browser (SSB). PWAs being run on an SSB provides a similar experience to using native applications (Osmani, 2015). Figure 11 is a comparison between how a PWA is viewed in a normal web browser and an SSB.

29

Figure 11: ReactJS PWA on a Chrome browser window (left) and a Chrome SSB Window (right) Unlike native applications, a PWA is distributed by visiting a website and directly within the interface of the web browser, making the conscious decision to “install” that website. When a PWA is installed, it becomes available to function without an internet connection. Every browser has its own criteria for the installation of PWAs. For a PWA to be installable on a Chrome web browser it must satisfy three requirements (LePage, 2021). The browser looks for these requirements before allowing the user to install the PWA. The requirements are: ● Having a Service Worker listening for fetch events ● Serving the page on an HTTPS connection ● Having a Web app manifest file It was planned to enforce a fourth requirement for Chrome 93, this requirement entailed an offline fall-back page if the device is not connected to the internet. Google has now revoked that requirement for future Chrome releases and put it on hold (McLachlan, 2021). Service workers are JavaScript files run in the background. For a PWA, a Service worker usually handles offline functionality. Another property of a PWA is to have the capability to access the hardware of its host devices such as the camera, microphone, and geolocation. Sending and receiving push notifications is another capability that PWAs have. While these features are not specific to a front-end framework, it is closely tied to PWAs and the near-native experience they can provide (Richard & LePage, 2020). Currently, the development of the PWA front-end is done with either the use of JavaScript frameworks or plain JavaScript. However, the latest implementation of JavaScript is considered insufficient for the development of high-performing web

30

applications (Haas, et al., 2017; Price & Eberhardt, 2018), meaning that there is a need for an alternative that can do heavy computations with minimal overhead.

3.2.2 WebAssembly WebAssembly is a low-level instruction language by the WebAssembly Working Group. The group contains members from 19 organizations including Mozilla, Microsoft, Google, and Apple. The language has been approved as a web standard by the World Wide Web Consortium (W3C). WebAssembly aims to complement JavaScript and fulfil low-level tasks that JavaScript subsidiaries like asm.js have failed to satisfy effectively (Haas, et al., 2017). Due to WebAssembly being a W3C standard, it is supported by Firefox, Safari, and Chromium browsers (Edge, Chrome, Opera, etc.). WebAssembly is partly designed to be a compilation target, making it possible to write applications in programming languages other than JavaScript and make them executable in the browser. JavaScript and WebAssembly use the same interpreter engine when executed. What differentiates the two are the steps that are taken to parse their code to bytecode. Every major interpreter engine handles its source code differently, but Clark (2017) has made a comparison in how SpiderMonkey could run WebAssembly code in comparison to JavaScript. JavaScript has two extra steps in its parsing chain, these steps are garbage collection and re-optimising. Microsoft introduced the Blazor WebAssembly framework to replace JavaScript front-end code with their own .NET platform (Microsoft, n.d.).

3.2.3 Virtual DOM Before describing the technical concepts around Blazor WebAssembly, one of the major patterns needs to be outlined. Namely the Virtual DOM. To alter the layout of a page during runtime, it is required to use the DOM API found in the web standard. But implementing efficient JavaScript that alters DOM efficiently takes development time and requires very specific layout-altering. To simplify front-end development a new approach was developed, this pattern is called Virtual DOM (VDOM). Figure 12 presents an overview of how a page layout can be altered using the VDOM approach.

31

Figure 12: Overview of a Virtual DOM when changing an element VDOM is a method that is an in-memory representation of the current state of the page, this is represented as a data tree. When the in-memory representation is changed a new tree is generated. A differentiation algorithm (or diff algorithm) is used between the old and new trees to track which changes have been made to what element. From the diff algorithm, a set of instructions are generated, these are used for executing layout altering functions on the DOM API.

3.2.4 Blazor WebAssembly Blazor WebAssembly (Blazor Wasm) is a Single-Page Application (SPA) framework developed by Microsoft. Blazor Wasm can be used to create SPA with PWA characteristics.

32

The source code in a Blazor Wasm is written in C#. By using a WebAssembly runtime called Mono, a Blazor Wasm application can run its front-end code on the browser. Mono is a .NET runtime and is used by Blazor Wasm to execute compiled .NET code on the web browser. A Blazor Wasm application uses the Mono runtime as a compilation target. Compiling the runtime and letting it interpret the source code during runtime is called Interpreted mode. Meaning that C# code only needs to be compiled for the .NET runtime to work in the browser. An overview of how the architecture of a Blazor Wasm application that is compiled in Interpreted mode is seen in Figure 13.

Figure 13: Sanderson, S. (2018). Interpreted Mode Overview [Figure]. Retrieved 15 April 2021, from https://blog.stevensanderson.com/2018/02/06/blazor-intro/ Front-end code in a Blazor Wasm application is characterized by user interface (UI) logic. UI Logic refers to how the state of the interface changes depending on pre- defined logic and user input. A web page that changes its content in this manner is referred to as a dynamic page. Dynamic pages are web pages that change depending on the current state of present data. Normally the state of a dynamic page is determined by the hosting server but in Blazor Wasm’s case, the state is determined by the client. Razor is a markup language that combines HTML syntax with C#, enabling the developer to use C# syntax for generating the template. When using Razor markup in Blazor it binds data with the client, builds the page, and changes the layout depending on logic written in C#. In Blazor Wasm, all pages and page-specific logic are written in Razor markup within component files. To be resource-efficient in altering the page layout, the Blazor Wasm client only updates the elements whose state gets modified. This is done using a VDOM

33

implementation called RenderTree. RenderTree is used to store the current DOM in a tree structure. When the layout of the page needs to change, Blazor generates a new RenderTree that is then compared to the old one. By using the differentiations found between the two RenderTrees, a UI Differentiation is created which is a set of instructions on how the DOM should be updated (Himschoot, 2019). An overview of RenderTree objects and UI-events is seen in Figure 14.

Figure 14: Roth, D., Fritz, J., Southwick, T. (2020). Overview of the flow for a RenderTree to DOM-instructions [Figure]. Retrieved 15 April 2021, from https://raw.githubusercontent.com/dotnet-architecture/eBooks/master/current/blazor- for-web-forms-developers/Blazor-for-ASP-NET-Web-Forms-Developers.pdf

3.2.4.1 C# Timer class The Timer class is a C# class that handles events fired in an interval. It is a standard class included in .NET. On a finished interval, the Timer class invokes an asynchronous method which in the experiment application is named OnTimedEvent. Even if the timer class is labelled as asynchronous it still runs on one thread due to the WebAssembly standard being executed on one thread. When analysing gathered Blazor Wasm data from DevTools, short JavaScript calls before and after each TimerFire event were noticed. In a Blazor Wasm application, the timer fires twice per interval. The first time it fires is to set the timeout parameter and the second is executing the programmed instructions. In the Mono runtime, this function is called _mono_set_timeout. When measuring the amount of time, it takes to generate a VDOM tree the first call is ignored by linking the TimerFire event with the origin function (_mono_set_timeout).

34

3.2.5 ReactJS ReactJS is a SPA framework used for front-end development in web development. JavaScript eXtension (JSX) and the VDOM are the main features that React offers. JSX needs to be compiled to JavaScript upon application build. JSX allows modification and updating of the DOM through HTML-style code along with JavaScript (ReactJS Contributors, 2020). Unlike Blazor Wasm, ReactJS uses JavaScript for its front-end, meaning that there is no compilation step before executing the code. Every browser that runs JavaScript and WebAssembly interprets the source code in its own interpreter engine during runtime. In most Chromium-based web browsers, JavaScript and WebAssembly are interpreted by the V8 Engine and goes through a set of processes before becoming machine code.

3.2.6 V8 JavaScript Engine V8 is the JavaScript engine used in Google Chrome, it is open source and started development by engineers at Google. A JavaScript engine generally handles the parsing and execution of JavaScript. The responsibilities of V8 include memory allocation, garbage collection and the compilation and execution of JavaScript and WebAssembly. One aim of V8 is to compile source code to run as optimised as possible. To provide optimised code, it uses one of two pipelines. What pipeline is used is dependent on what standard is being compiled and executed. When compiling JavaScript, the source code is parsed to a tree structure. This tree structure takes every condition and operation to construct an Abstract syntax tree (AST). After generating the AST, it is passed to Ignition (McIlroy, 2016). This is done only once. Ignition is an interpreter that converts the AST to bytecode, this first pass is not fully optimised and is therefore passed to TurboFan. TurboFan is an optimising compiler that takes bytecode and optimises it in iterations, this is done by reducing complexities found in the bytecode. When TurboFan is done it replaces parts of the earlier bytecode with optimised machine code (Hinkelmann, 2017). See Figure 15 for

35

an overview of how JavaScript source code is compiled.

Figure 15: Hinkelmann, F. (2017). V8’s compiler pipeline for JavaScript [Figure]. Retrieved 23 May 2021, from https://medium.com/dailyjs/understanding-v8s- bytecode-317d46c94775 WebAssembly is run by a different set of components, see Figure 16 for an overview of components and steps taken to compile a module. A WebAssembly module is first compiled by LiftOff, the LiftOff compiler is designed to compile WebAssembly fast. This is done by compiling every instruction in the module iteratively to machine code. After the first compilation in LiftOff, the module is optimised. This is done by providing the module to TurboFan in a background thread, when a function is optimised, it replaces the corresponding function found in the resulting machine code (Chromium Contributors, 2020; Backens, 2018).

36

Figure 16: V8 WebAssembly compilation pipeline overview However, TurboFan does not optimise everything that executes during runtime. Code that is executed often is coined as hot code. When hot code is detected, it is passed to TurboFan for recompiling and optimisation procedures. On the other hand, when code is not executed frequently it is called cold code and will not be selected as frequently by TurboFan for optimisation (Guo, 2020).

37

4 Results

In this chapter, the collected data and the obtained results are presented. These are further analysed using the analysis methods described in 2 Method and Implementation.

4.1 PWA support for each framework For each framework, the official documentation and the official YouTube channels have been used as sources to gather data and provide an answer for RQ 1. The collected data for each framework is presented separately in the subjects defined in 1.4 Scope and Limitations. Table 4 presents which keyword is mentioned in which source of information. A red cell indicates that it was not mentioned while a green cell indicates that it was mentioned.

38

Important aspects ASP.NET dotNet ReactJS Facebook for PWA Documenta YouTube Documentati Dev. tion Channel on Youtube

Offline/Cache mode

Caching pattern

Background updates

IndexDB

Manifest

Push notifications

Service Worker

Hardware access

Camera & Microphone

Geolocation

Vibration (mobile)

Device orientation & motion

Not mentioned Mentioned

Table 4: Keywords mentioned in each documentation source

4.1.1 Blazor WebAssembly This section presents a summary of what information is mentioned for each PWA keyword in the Blazor Wasm sources. Offline/Cache first The official YouTube channel DotNet (stylised dotNet) has numerous videos published where the creation of PWAs using the Blazor Wasm framework is presented. Many of the videos are tutorials that emphasize the importance and responsibilities of Service Workers (dotNET, n.d.).

39

The documentation also clearly educates the developer about the complications that come along with offline-mode and when the usage of offline-mode might be inappropriate. This is done by listing the advantages and features that are provided by the offline mode to let the developer decide upon the appropriateness of using offline support (Anderson et al., 2021). Caching pattern The YouTube content mentions that cached content is returned, regardless of access to the network or if newer content is available on the server. Using hashing for each file, the Service Worker ensures that the snapshot of the resource is complete and self- consistent. Regardless of the availability of a newer version, the cached snapshot is sufficient for a successful interaction between the user and the application (dotNET, 2020). Background updates Further, it is mentioned that the Blazor PWA template enables automatic background updates granted that a network connection is available, and the Service Workers are used in the application. Upon an online visit, the cached files are compared, and the Service Worker updates itself to create a new cache and save the new version of the resources. Upon failure, the process is repeated on the next online visit. For completion of an update, it is required for the user to navigate away and close the application completely for the new Service Worker to activate (Anderson et al., 2021). IndexDB According to the tutorial on Blazor Wasm PWA, an asynchronous method can be used to store data in the browser using IndexDB. This is accomplished by using the IndexDB API (dotNET, 2020). Push notifications Blazor Wasm supports receiving and sending push notifications but is independent of the framework. The backend server is responsible for sending. Receiving and displaying a notification on the client is implemented in the Service Workers. Implementation of the Notification API asks the user for permission and registers the subscription for notifications. Thereby, the push notifications are available in offline mode (dotNET, 2020). Service Workers The documentation articulates that registration of the Service Workers occurs as an initial step in the project and is crucial for offline support. They enable offline mode

40

by caching the content on the first online visit by the user, and stores resources in the local memory. The Service Worker updates the cached content on the next online visit. This implies that the developer lacks control of what version the user is using unless they have completed an update by making a new visit. Accordingly, the user's cached version can be any historical version. This becomes an issue if the older versions are not compatible with the recent changes of the application (Anderson et al., 2021). 4.1.2 ReactJS This section presents a summary of what information is mentioned for each PWA keyword in the ReactJS sources. Offline/Cache-first mode According to the documentation, ReactJS’s ability to develop offline-first PWAs is granted with the use of Service Workers and Workbox Webpack which handles the pre-caching. The Workbox Webpack plugin is necessary for granting the application offline capabilities (Timothy, 2020). Caching pattern The caching of the content is done by the Service Workers in the same manner as Blazor Wasm. On their YouTube channel, ReactJS discusses the usage of the Webpack plugin which prevents unchanged files from being cached unnecessarily (Facebook Developers, 2017). According to ReactJS conferences, the aggressive caching method aids the PWA to achieve the speed that matches the start-up time of native mobile applications (Facebook Developers, 2016). Background updates Similarly, to Blazor Wasm, the updated Service Worker awaits the user re-visiting the application before updating the content in order to preserve the speed and reliability. This brings the advantage of faster navigation throughout the application. However, this also implies that the version the user is utilizing is dependent on the last update of the Service Worker (Timothy, 2020). Manifest The manifest file is created at the start of a project and the developer has full control over the logic presented there (Timothy, 2020).

41

Push notifications As stated in the documentation, the push-notification library can be added directly to the manifest file. Registered Service Workers utilize the API for push notifications to send and receive notifications (Timothy, 2020). Service Workers The Service Workers are an essential part of a PWA due to their responsibilities for offline capabilities. According to the documentation, these are registered in the manifest file of the application and thereby used through the platform (Facebook Developers, 2016).

4.2 Experiment Sessions A total of 5392 data points were collected for the experiment. All three ReactJS measurements provided 900 points with each while the Blazor Wasm measurements varied between 873 and 930. See Table 5 for plotting of collected data points and sessions.

Sample Session 1 points Session 2 points Session 3 points

Blazor Wasm 873 889 930

ReactJS 900 900 900

Table 5: Amount of data points gathered for the experiment Each file contains rows holding a value for tree generation time, tree size in memory and fraction spent on layout generations of total time. All results are rounded to the nearest third decimal.

4.2.1 Session 1 - 720 pairs In this session, the set number of generated pairs is 720. The calculated means for each variable within each sample are seen in Table 6. The standard deviation for each variable is viewed in Table 7.

42

Framework Generation time Tree size mean Generation mean (μs) (byte) Partition mean (% of total time spent on generating trees)

ReactJS 496463,014 360071,858 0,03

Blazor Wasm 189313,085 17999,368 0,03

Table 6: Mean values gathered from session 1

Framework Generation time Tree size Generation Standard Standard Partition Deviation (μs) Deviation (byte) Standard Deviation (% of total time spent on generating trees)

ReactJS 55685,040 5751612,380 0,004

Blazor Wasm 29218,556 498313,507 0,005

Table 7: Standard deviation values gathered from session 1

4.2.2 Session 2 - 360 pairs Here the set number of generated pairs is 360. The means for each variable within each sample are seen in Table 8. The standard deviation for each variable is viewed in Table 9.

Framework Generation time Tree size mean Generation mean (μs) (byte) Partition mean (% of total time spent on generating trees)

ReactJS 305557,788 277851,040 0,033

Blazor Wasm 121891,294 27307,834 0,030

Table 8: Mean values gathered from session 2

43

Framework Generation time Tree size Generation Standard Standard Partition Deviation (μs) Deviation (byte) Standard Deviation (% of total time spent on generating trees)

ReactJS 61633,000 3365782,106 0,007

Blazor Wasm 31244,526 363692,796 0,009

Table 9: Standard deviation values gathered from session 2

4.2.3 Session 3 - 180 pairs Here the set number of generated pairs is 180. The standard deviation for each variable is viewed in Table 10. The means for each variable within each sample are seen in Table 11.

Framework Generation time Tree size mean Generation mean (μs) (byte) Partition mean (% of total time spent on generating trees)

ReactJS 124339,028 214854,573 0,03

Blazor Wasm 63010,237 28425,787 0,03

Table 10: Mean values gathered from session 3

Framework Generation time Tree size Generation Standard Standard Partition Deviation (μs) Deviation (byte) Standard Deviation (% of total time spent on generating trees)

ReactJS 55304,532 1674383,332 0,015

Blazor Wasm 22084,260 272603,585 0,011

Table 11: Standard deviation values gathered from session 3

44

4.3 Data Analysis A Z-test is conducted on gathered data from the experiment. Data from the theoretical literature study is analysed.

4.3.1 RQ 1 - Literature analysis The results of this study imply that the official documentation of the Blazor Wasm and the ReactJS framework is different in means of providing in-depth information about their support for PWA development.

4.3.1.1 Analysis of results for Blazor Wasm The Blazor Wasm documentation is thoroughly written and detailed, providing the reader with in-depth information about each subject. It is also ensured that the reader understands that the subjects are specific to the PWA concept and not impacted by the framework chosen for its development. Further documentation and tutorials of how PWA can be developed using the Blazor Wasm framework are available on dotNET YouTube channel. In the published videos, developers tutor the viewer on the necessary precautions to take when creating a new PWA project. Throughout the documentation, the reader is introduced to the terms and aspects that are specific for PWAs and given recommendations and notified about caveats. There is a template project used as a starting point for a PWA. Otherwise, there are no external PWA tools provided for PWA development according to the documentation. The reader is also familiarised with Service Workers, offline-mode, caching patterns, local storage of data, and background updates. Despite the detailed documentation, there are no mentions of the hardware access and whether there is any extra functionality provided by Blazor Wasm in this area.

4.3.1.2 Analysis of results for ReactJS The documentation for ReactJS covers the initial steps that the reader needs to take to start developing a PWA. This includes an introduction to and usage of Service Workers, offline mode, the manifest file, and push notifications. However, there are components that are not explained in the documentation and the reader is referred to other sources. These are external sources such as blog posts belonging to developers at Google and articles from Google Developer website. The Facebook Developers’ YouTube channel does not provide tutorials on the creation of PWA projects using ReactJS. Most of the videos are of conferences led by developers with PWAs. Moreover, these videos are outdated and fail to provide new information compared to the documentation on the website.

45

Like the Blazor Wasm documentation, ReactJS also recommended the reader restrain from using the offline mode in the testing and development stage of the application. Nor should it be used if the PWA does not have use cases that require offline capabilities. Missing in the ReactJS documentation is the usage of IndexDB despite its importance for local storage of the data within the application. Moreover, the documentation does not mention the PWAs capability to access the hardware of the device it is installed on. Nor is there any mention of any external PWA tools provided by the framework.

4.3.2 RQ 2 - Experiment analysis

Table 12 views the results from the Z-test. Red indicates proven null hypothesis H0 while green indicates proven alternative hypothesis H1. Green cells prove that there is a significant difference between each mean value. Red cells indicate the opposite, meaning that there is no statistically significant difference between the compared samples, indicating that the difference between the two sample means is a coincidence.

Session number zgen. time ztree size zgen. use

1 (720 Pairs) 79.638 2.193 -832.597

2 (360 Pairs) 147.560 1.777 -5.018

3 (180 Pairs) 30.964 3.293 1.744

퐶푟푖푡푖푐푎푙 푣푎푙푢푒 Reject H1 if Reject H0 if otherwise ≈ 1,95996 -1,960 < z < 1,960

Table 12: Z-test results Two out of nine means are not statistically significant. This indicates that for session 2 the mean tree size between the Blazor Wasm PWA and ReactJS PWA could not be concluded as statistically different. The same goes for the mean value in portion of generation during session 3. 4.3.2.1 RQ 2.1 - Time difference [2.1] How much time does it take to generate a VDOM tree between the two frameworks? The differentiation gets higher the more elements each framework must take care of as seen in Table 13.

46

Session Difference in mean tree Longest mean tree generation time (μs) generation time

1 (720 pairs) 307149,930 ReactJS

2 (360 pairs) 183666,494 ReactJS

3 (180 pairs) 61328,791 ReactJS

Table 13: Tree size mean difference and framework with largest mean The ReactJS PWA had the longest generation times in every scenario and its loading times increased much faster than the loading times found in the Blazor Wasm PWA. This pattern is seen in Figure 17.

Figure 17: Generation time means line graph 4.3.2.2 RQ 2.2 - Tree sizes [2.2] What are the size differences of VDOM trees in memory between Blazor Wasm and ReactJS? Difference in the mean tree size is seen at Table 14. Blazor Wasm used the least mean memory when after generating trees during every session. ReactJS consumed more mean memory the more elements it had to handle. This makes the difference in mean tree size shrink from session 1 to session 3.

47

Session Difference in Difference in Framework with mean tree size mean tree size largest tree size (%) (byte) mean

1 (720 pairs) 180,957 342072,490 ReactJS

2 (360 pairs) 164,205 250543,206 ReactJS

3 (180 pairs) 152,847 186428,786 ReactJS

Table 14: Tree size mean difference and framework with largest mean The downward trend from session 1 to session 3 in ReactJS is viewable in Figure 18. See Figure 19 for the upward trend found in the Blazor Wasm results.

Figure 18: Tree size means line graph

48

Figure 19: Blazor Wasm tree size mean 4.3.2.3 RQ 2.3 - Consistency in Loading Times [2.3] How consistent is each VDOM implementation in generation times? Consistency is defined as standard deviation. The lower the standard deviation is the less spread of the gathered values. A line graph viewing the calculated standard deviation between both frameworks during the three sessions can be viewed in Figure 20.

49

Figure 20: Portion of Time Spent Loading Standard Deviation There is a difference between the standard deviation values from each session. How many percent different each standard deviation is viewed at Table 15.

Session Standard deviation Framework with highest difference (%) standard deviation

1 (720 pairs) 25,007 Blazor Wasm

2 (360 pairs) 29,924 Blazor Wasm

3 (180 pairs) 27,002 ReactJS

Table 15: Percentage of difference between standard deviations Blazor Wasm is more inconsistent compared to ReactJS when handling many elements. However, when handling 180 pairs the Blazor PWA proved to be more consistent in its loading times. The overall trend in both frameworks indicates less consistency in generation times when handling a small number of elements and more consistency when handling many elements.

50

5 Discussion

In this section, the gathered results are discussed. What the results implicate and why certain trends are observed. How the chosen method worked for the results is also discussed. These two discussions are split up into their own sections.

5.1 Results discussion The purpose of the study is as stated in 1.3 Purpose and research questions: To evaluate the differences that the Blazor Wasm framework can yield for PWA front- end development compared to the ReactJS framework. In this study, two PWA experiment applications were created and measured with the goal of measuring VDOM generation time, tree size and the portion of total time to generate. The gathered data has then been compared with ReactJS as a baseline and has created a baseline for the Blazor Wasm framework in VDOM generation. However, this study is limited to two aspects. The first aspect is VDOM generation times and memory usage in analysing the performance. The second aspect is what Blazor Wasm and ReactJS yields outside of the framework is limited to official documentations only.

5.1.1 PWA Support in the Blazor Wasm documentation [1] To what degree does Blazor Wasm support PWA development in comparison to ReactJS? The results of this literature study suggest that information provided in the frameworks’ documentation from the selected sources is lacking in various areas. When compared to ReactJS, the Blazor Wasm framework documentation is presented with extensive clarity and detail. It was also observed that ReactJS refers to external resources where the reader can access more detailed information about the subjects. Meanwhile, the Blazor Wasm documentation provides explanations of every subject presented in the article. This is done through figures, articles, and code snippets. Another differentiating factor is the date on which the content has been published. The ReactJS YouTube content about PWA’s is older and not closely related to the ReactJS framework. In contrast, the Blazor Wasm YouTube content is more up to date and thorough. The content makers also make use of the framework when demonstrating how PWAs can be developed. Common between the two is the presentation of the offline mode capabilities, caching patterns, manifest, and push notification ability. The two documentations also extensively explain the importance and responsibilities of Service Workers. What is

51

missing in each documentation is the mentioning of hardware access. Neither Blazor Wasm nor ReactJS provides any information on how this can be utilized. The obtained results indicate that both frameworks support the development of PWAs. While some PWA specific features are clearly mentioned, others are not acknowledged in the documentations. However, the PWA specific features are not dependent on the framework. Most of these are based on APIs and only require support for the usage of the API to achieve the intended outcome. One of these features is the Service Worker API which is crucial for PWAs to function offline. Another observation is the publishing date of the information for each framework. The ReactJS documentation indicates that presented information can be outdated and published prior to the discovery of certain PWAs features. One interpretation of this can be that ReactJS has this support, but the documentation fails to acknowledge this due to being outdated. Another interpretation is that the feature is independent of its framework and thereby not mentioned. Neither of the interpretations indicates that ReactJS would be less appropriate for PWA development. In contrast, the Blazor Wasm documentation is recently updated, and the published video content is more detailed and closely related to the framework. The obtained result can be interpreted as a strategy to actively promote the framework by bringing forward the features it supports. Another interpretation can be that the documentation is more dedicated to achieving an educational outcome.

5.1.2 Virtual DOM differentiation [2] How does a Blazor Wasm application differentiate itself from established JavaScript frameworks in VDOM generation performance? For this research question and its subset, two experimental PWAs were developed with the goal of triggering VDOM generation from each framework. A Z-test was conducted on every mean value to test the difference between each sample. From two samples it was proven that there was a statistical difference between 7 out of 9 mean values. This indicates that there is a statistical difference between the two VDOM implementations found in Blazor Wasm and ReactJS. [2.1] How much time does it take to generate a VDOM tree between the two frameworks? In this study, a faster VDOM generation was observed in the Blazor Wasm PWA when compared to the ReactJS PWA. About 61,9%, 60,1% and 49,3% decrease in generation times on the Blazor Wasm PWA was observed. This is in the range of

52

previous execution time measurements by Reiser & Bläser (2017), Haas et al. (2017) and Wang et al. (2019). With this knowledge, an indicator for [2.1] is provided. Drawing from papers by Reiser and Bläser (2017), Haas et al. (2017) and Wang et al. (2019), the WebAssembly standard is one factor in the lower generation times between ReactJS and Blazor Wasm. This increase of speed could be due to the invocation of WebAssembly modules taking less steps in comparison to JavaScript (Clark, 2017). This could also be an indicator that the interpreting strategy of LaunchPad is more speed efficient than Ignition in the V8 engine. [2.2] What are the size differences of VDOM trees in memory between Blazor Wasm and ReactJS? Changes in memory usage were analysed from each application. The memory changes are recognised as VDOM tree sizes and were used to compare the average VDOM tree size between each framework. It was shown that the Blazor Wasm PWA compared to the ReactJS PWA used 0,34 MB, 0,25 MB and 0,18 MB less mean memory when generating VDOM trees during each session. This difference value is smaller than what was observed between StdWeb, Yem and ReactJS in the benchmarks by Krause. (2018) But the difference in percentage is much larger. See Table 16 for the comparison with the replace benchmark, the percentages are rounded to the nearest integer.

StdWeb - React 16.4.1 Yem - React 16.4.1 Blazor Wasm (.NET 4.0) Replace benchmark Replace benchmark - React 17.0.2 Session 3 Memory difference (%) Memory difference (%) Memory difference (%)

59 39 152

Table 16: Percentage difference between gathered mean with the smallest difference and replace benchmark results from Krause The ReactJS mean tree size increased expectantly when handling a larger number of elements but the mean tree size in Blazor Wasm decreased. Which is an unexpected pattern. It is suspected that the method of gathering VDOM tree size from the Blazor Wasm profiles fails to recognize the memory allocation pattern for Blazor Wasm applications. This issue is discussed in 5.2.1 When Analysing Memory From the Experiment. The results need further analysis to be a conclusive answer for [2.2]. [2.3] How consistent is each VDOM implementation in generation times?

53

Consistency in how long it takes to generate a VDOM-tree was measured by getting the portion of the total amount of generation times and then calculating the standard deviation for each session. The findings conclude that Blazor Wasm was less consistent in its loading times in all sessions except session 3. In session 3, the ReactJS PWA spiked in inconsistency and surpassed the Blazor Wasm PWA. Worth noting is that the Blazor Wasm PWA also increased in inconsistency during this session, indicating a general pattern between the two standards. Why there is an inconsistency spike at session 3 (180 pairs) could be that TurboFan has not had as many optimisation-runs as the previous sessions. The ScrambleTree function in both implementations is invoked respectively 360- and 720 times during the other sessions, indicating a threshold between 180- and 360 times where major optimisations are done to the bytecode. For the remaining sessions, it could be an indicator of a more efficient code base on ReactJS. There could also be performance issues with having a runtime between source code and interpreter for the Blazor Wasm PWA. This observed pattern concludes that the fewer elements a PWA handles, the more inconsistent its VDOM generation times become. Therefore, the difference in consistency is observed and provides part of an answer for [2.3]

5.2 Method discussion To provide answers for RQ 1, the literature study is selected as the appropriate method. The obtained results fulfil this purpose. For the legitimacy of the study, the sources for information collection are limited to ReactJS and Blazor Wasm official documentation along with the official YouTube channels for these two frameworks. This can weaken the method as limited sources lead to limited information. However, it can be argued that the two companies are responsible to keep the documentation up to date and detailed. Official documentations are usually authored by people who obtain knowledge about the inner workings of a product and should thereby be the primary source of information for its user. Hence, it can be justified to expect the official documentation to provide clear and detailed information. The method is designed to search for PWA specific aspects supported by each framework. Analysing the obtained results indicates that these aspects are independent of the framework as they use APIs to achieve the desired functionality. Since both frameworks support the usage of APIs, it is an indication of equal support for PWA development along with its specific features. However, the amount of information

54

about these features in each of the frameworks’ documentation can determine the developer’s selection of the framework. A different approach can be taken to further fulfil the purpose. A survey can be conducted on a selected group of developers with the purpose to evaluate their experience using the two frameworks. This can provide a clearer indication of how well the frameworks are suited for PWA development by analysing the developers’ experience with each framework. For the experiment, using the Chrome DevTools proved to be a good choice for this experiment. It provided insightful information regarding sources of function calls, this aided in identifying the extra amounts of TimerFire calls that the Blazor Wasm application invoked. However, using Chrome DevTools limits this experiment to be measured to Chromium-based browsers only. If it were to be repeated on another browser, the choice of measuring instrument would have to be reconsidered. The experiment also took a long time to conduct with each instance taking more than 30 seconds to finish. This led to very long experiment sessions when testing up to 60 instances per session. An alternative to shortening the experiment time would be an experiment application that does not rely on events occurring during an interval and the collection of data being conducted by a headless solution like Persson (2020) used in his test case.

5.2.1 When Analysing Memory From the Experiment When examining heap usage changes in each framework the same pattern of heap allocation was expected. The method of gathering the relative heap change is explained in 2.2.3 Measuring Performance in Chrome DevTools. This method seems to work well for the ReactJS application where memory is allocated a few microseconds after the JavaScript portion. But this method gives unexpected results when applying the same method when analysing data from the Blazor Wasm PWA. The Blazor Wasm tree mean size gets smaller the more elements it handles in the analysis. It can be concluded that this seemingly increase in error is correlated with a gap pattern found in the timeline when inspecting a randomly selected profile via the DevTools interface. As seen in Figure 21, the gap gets larger the more pairs of elements the session handles. The figure is three runs from each session. Each represents a timeline in heap usage from left to right, the higher line the more memory is used.

55

Figure 21: Blazor Wasm memory allocation at the end of a TimerFire event. Session 1: 720 pairs (left), session 2: 360 pairs (middle) and session 3: 180 pairs (right) This gap is placed about 10 microseconds after the execution of TimerFire and the memory differences from the bottom of the gap to the continuation ledge is from left to right 391 916, 160 228 and 118 212 bytes. The gap is not found after every allocation either, making it more difficult to add an exception for the analysis of Blazor Wasm PWA data.

56

6 Conclusions and further research

Conclusions are made and motivated in this section based on the results obtained by the literature study and the experiment. Further outlooks based on found data is also outlined.

6.1 Conclusions Based on the obtained results and the outcome of the analysis, it is concluded that Blazor Wasm and ReactJS have documentations that differ when regarding content. Further, it is concluded that Blazor Wasm VDOM provides faster loading times but shows inconsistency when compared to ReactJS.

6.1.1 Blazor WebAssembly as a PWA framework The acquired results do not provide a clear indication of the PWA methodology being clearly related to any framework. Based on the obtained results it can be concluded that the PWA features are only dependent on the APIs and browser support, while independent of the implementations of the framework used for its development. It can be concluded that the PWA specific features are not related to the frameworks. Also, no external tools are provided by either company in aiding PWA development. However, the written- and video documentation of the Blazor Wasm framework covers more PWA features when compared to ReactJS. If PWA support is defined by the extent it documents the methodology and its subjects, then Blazor Wasm provides more support for PWA development than ReactJS. Like the study conducted by Biørn-Hansen et al. (2018a), the PWA methodology is not dependant on a specific framework. This can be due to the PWA features using APIs for functionality, which is unrelated to the framework.

6.1.2 Blazor WebAssembly and VDOM Generation The tree size measurement was inconclusive and requires further measurements with adjustments to how heap memory changes are analysed in a Blazor Wasm application. However, the data gathered for generation times and loading time consistency proved to be useful and can be used to provide data for the problem statement regarding the lack of documentation around WebAssembly VDOM generation performance. Concluding that problem statement, Blazor WebAssembly proves to be a faster framework in VDOM generation in a PWA environment provided by Google Chrome when altering large amounts of nested elements. Additionally, ReactJS is more consistent when handling a larger number of elements but gets more inconsistent when handling a small number of elements in comparison to Blazor Wasm.

57

6.1.3 Practical implications In this section, the practical implications deferred from the results of this study are listed. Practicality is defined by the area outside of science. ● That Blazor Wasm is a much faster framework when it comes to changing large rows of HTML elements ● ReactJS documentation regarding PWA development is much more dependent on external resources than the Blazor documentation ● ReactJS PWA content on YouTube lacks detail and is outdated ● Documentation and video content for the Blazor framework covers more PWA topics than the resources found on ReactJS ● The outcomes of this study can guide developers in their selection of framework when developing a PWA

6.1.4 Scientific implications In this section, the scientific implications deferred from the results of this study are listed. ● Methods and considerations in analysing memory changes in Blazor Wasm and ReactJS applications ● A baseline in observed loading times, tree sizes and loading consistency to defer VDOM research from ● The fewer elements both the ReactJS and Blazor Wasm framework has to handle, the less consistent is the loading times

6.2 Further research There are some major directions to further this area in terms of using retrieved results as a baseline or apply the same method on other frameworks/platforms. Along with the release of .NET 6 comes an Ahead of Time (AoT) runtime for Blazor applications. AoT infers that more parts of the source code are precompiled and could therefore improve performance according to Roth (2020b) in a comment response on his blog post announcing the release of Blazor WebAssembly 3.2.0 Preview 4. At the time of writing, a preview version of the AoT runtime has been released for Blazor. This runtime could be put under scrutiny regarding VDOM generation. The experiment was only conducted on one platform and one web browser. With the PWA methodology being presented as a cross-platform solution it is interesting to measure the performance in Blazor Wasm on alternative platforms. Regarding the lower generation times and memory usage gathered from Blazor Wasm, the framework could be scrutinized further on platforms with weaker hardware like smartphones, smart TVs, or smartwatches.

58

7 References

Altexsoft. (2018, May 10). Progressive Web Apps: Core Features, Architecture, Pros and Cons. Retrieved from https://www.altexsoft.com/blog/engineering/progressive-web-apps/ An, D. (2018, February). Find out how you stack up to new industry benchmarks for Mobile Page Speed. Think with Google. Retrieved April 4, 2021, from https://www.thinkwithgoogle.com/intl/en- 154/marketing-strategies/app-and-mobile/how-stack-up-new-industry- benchmarks-for-mobile-page-speed/ Anderson, R., Addie, S., & Luke Latham. (2021, January 11). Build Progressive Web Applications with ASP.NET Core Blazor WebAssembly. Microsoft Docs. Retrieved Mars 10, 2021, from https://docs.microsoft.com/en-us/aspnet/core/blazor/progressive-web- app?view=aspnetcore-5.0&tabs=visual-studio Backens, C. (2018, August 20). Liftoff: a new baseline compiler for WebAssembly in V8 · V8. V8 Blog. Retrieved May 24, 2021, from https://v8.dev/blog/liftoff Biørn-Hansen, A., Grønli, T.-M., & Ghinea, G. (2018a, November). 108A Survey and Taxonomy of Core Concepts and Research Challenges in Cross-Platform Mobile Development. ACM Computing Surveys, 51(5), 34. 10.1145/3241739 Biørn-Hansen, A., Majchrzak, T. A., & Grønli, T.-M. (2018b, June). Progressive Web Apps for the Unified Development of Mobile Applications. 10.1007/978-3-319-93527-0_4 BlinkOn. (2020, December 1). Life of a Script [Video]. YouTube. https://youtu.be/veYjbF1rt5o Chrome Team. (2021, June 26). Fugu API Tracker. Fugu Tracker. Retrieved June 26, 2021, from https://fugu- tracker.web.app/?version=0&search=&platforms=%5B%22pwa%22% 5D&bug=

59

Chromium Contributors. (2020, November 9). WebAssembly compilation pipeline · V8. V8 Docs. Retrieved May 24, 2021, from https://v8.dev/docs/wasm-compilation-pipeline Clark, L. (2017, February 28). What makes WebAssembly fast? Mozilla hacks - the Web Developer blog. https://hacks.mozilla.org/2017/02/what- makes-webassembly-fast/ DevelopIntelligence. (2018, April). 2018 DI Developer Survey Report. https://www.developeracademy.io/wp-content/uploads/2018/08/v2- 2018-DI_Survey-Report.pdf dotNET. (n.d.). Home [YouTube channel]. YouTube. Retrieved May 5, 2021, from https://www.youtube.com/dotnet dotNET. (2020, October 15). ASP.NET Core Series: PWA's with Blazor [Video]. YouTube. https://youtu.be/68pTZV5qcgE Facebook Developers. (2016, February 25). React.js Conf 2016 - Aditya Punjani - Building a Progressive Web App [Video]. YouTube. https://youtu.be/m2tvYGCdOzs Facebook Developers. (2017, March 16). [Emoticon] - Nicole Chung - How to React in the Slow Lane - React Conf 2017 [Video]. YouTube. https://youtu.be/pSr6sTRDdi4 Felderer, M., & Guilherme Horta, T. (2020). Contemporary Empirical Methods in Software Engineering (1st ed.). Springer International Publishing. https://doi.org/10.1007/978-3-030-32489-6 Google Chrome Developers. (2018, February 20). Progressive Web Apps – PWA Roadshow [Video]. YouTube. https://youtu.be/z2JgN6Ae-Bo Haas, A., Rossberg, A., Schuff, D. L., Titzer, B. L., Holman, M., Gohman, D., Wagner, L., Zakai, A., & Bastien, J. (2017, June). Bringing the web up to speed with WebAssembly. PLDI 2017: Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, 185-200. https://doi.org/10.1145/3062341.3062363 Himschoot, P. (2019). Blazor Revealed Building Web Applications in .NET. Apress. Hinkelmann, F. (2017, August 16). Understanding V8’s Bytecode. V8 is Google’s open-source JavaScript… | by Franziska Hinkelmann |

60

DailyJS | Medium. Medium. Retrieved May 23, 2021, from https://medium.com/dailyjs/understanding-v8s-bytecode- 317d46c94775 HM, H. (2020). PWA | 2020 | The Web Almanac by HTTP Archive. Web Almanac. Retrieved May 9, 2021, from https://almanac.httparchive.org/en/2020/pwa Huynh, D., & Nguyen-Ky, S. (2020, June 10). Engaging Building Automation Data Visualisation Using Building Information Modelling and Progressive Web Application. Open Engineering, 10(1), 434-442. https://doi-org.proxy.library.ju.se/10.1515/eng-2020-0054 Krause, S. (2018, September 27). Results for js web frameworks benchmark – round 8. Stefan_Krause.blog(). Retrieved May 16, 2021, from https://stefankrause.net/js-frameworks-benchmark8/table.html LePage, P. (2020, February 14). What does it take to be installable? Web.dev. Retrieved March 11, 2021, from https://web.dev/install-criteria/ Lewis, P. (n.d.). Rendering Performance | Web Fundamentals | Google Developers. Google Developers. Retrieved April 2, 2021, from https://developers.google.com/web/fundamentals/performance/renderin g/ McIlroy, R. (2016, August 23). Firing up the Ignition interpreter · V8. V8 Blog. Retrieved May 24, 2021, from https://v8.dev/blog/ignition- interpreter McLachlan, P. (2021, April 14). Improving Progressive Web App offline support detection - Chrome Developers. Chrome Developers. Retrieved May 22, 2021, from https://developer.chrome.com/blog/improved-pwa-offline-detection/ Mena, M., Corral, A., Iribarne, L., & Criado, J. (n.d.). Geonat. Geonat. Retrieved May 29, 2021, from https://app.acg.ual.es Mena, M., Corral, A., Iribarne, L., & Criado, J. (2019, July 31). A Progressive Web Application Based on Microservices Combining Geospatial Data and the Internet of Things. IEEE access, 7(1), 104577-104590. https://doi.org/10.1109/ACCESS.2019.2932196

61

Microsoft. (n.d.). Blazor | Build client web apps with C# | .NET. Microsoft. Retrieved February 15, 2021, from https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor Moerkerke, D. (n.d.). What PWA Can Do Today. Whatpwacando. Retrieved June 22, 2021, from https://whatpwacando.today/ Mozilla. (2021, May 7). Progressive web apps (PWAs) | MDN. MDN Web Docs. Retrieved from https://developer.mozilla.org/en- US/docs/Web/Progressive_web_apps O'Shannessy, P. (2013, July 2). Release v0.3.0 · facebook/react. GitHub. Retrieved May 11, 2021, from https://github.com/facebook/react/releases/tag/v0.3.0 Osmani, A. (2015, December). Getting Started with Progressive Web Apps. Google Developers. Retrieved February 28, 2021, from https://developers.google.com/web/updates/2015/12/getting-started- pwa Persson, M. (2020). JavaScript DOM Manipulation Performance: Comparing Vanilla JavaScript and Leading JavaScript Front-end Frameworks [Bachelor thesis]. http://bth.diva- portal.org/smash/record.jsf?pid=diva2%3A1436661&dswid=16189949 52882 Price, C., & Eberhardt, C. (2018). The Web Assembles: WebAssembly and the future of the web [White paper]. Newcastle upon Tyne: Scott Logic Limited. https://blog.scottlogic.com/ceberhardt/assets/white- papers/the-web-assembles.pdf Procida, C. (2017, February 22). claudiopro / react-fiber-vs-stack-demo [Github Repository]. Github. https://github.com/claudiopro/react-fiber- vs-stack-demo ReactJS Contributors. (2020, September 21). JSX In Depth - React. React. Retrieved May 29, 2021, from https://reactjs.org/docs/jsx-in- depth.html Reiser, M., & Bläser, L. (2017, October 24). Accelerate JavaScript Applications by Cross-Compiling to WebAssembly. VMIL'17: Association for Computing Machinery, 10-17.

62

https://concurrency.ch/Content/publications/Reiser_Blaeser_Accelerate _JavaScript_WebAssembly_VMIL_2017.pdf Richard, S., & LePage, P. (2020, January 6). What are Progressive Web Apps? Web.dev. https://web.dev/what-are-pwas/ Roth, D. (2020b, April 21). Re: Blazor WebAssembly 3.2.0 Preview 4 release now available [Blog comment]. ASP.NET Blog. Retrieved May 15, 2021, from https://devblogs.microsoft.com/aspnet/blazor- webassembly-3-2-0-preview-4-release-now-available/comment-page- 2/#comment-1919 Roth, D. (2020a, May 19). Blazor WebAssembly 3.2.0 now available | ASP.NET Blog. ASP.NET Blog. Retrieved May 11, 2021, from https://devblogs.microsoft.com/aspnet/blazor-webassembly-3-2-0- now-available/ Stack Overflow. (2020, February). Stack Overflow Developer Survey 2020. Stack Overflow. Retrieved February 17, 2021, from https://insights.stackoverflow.com/survey/2020#technology-web- frameworks-all-respondents2 Timothy. (2020, September 9). Making a Progressive Web App. Create React App. Retrieved May 8, 2021, from https://create-react- app.dev/docs/making-a-progressive-web-app/ Wang, S., Ye, G., Li, M., Yuan, L., Tang, Z., Wang, H., Wang, W., Wang, F., Ren, J., Fang, D., & Wang, Z. (2019). Leveraging WebAssembly for Numerical JavaScript Code Virtualization. IEEE Access, 7(1), 182711- 182724. 0.1109/ACCESS.2019.2953511 Web.dev Contributors. (2019, May 2). Avoid an excessive DOM size. web.dev. https://web.dev/dom-size/

63

8 Appendixes

Appendix 1 GitHub Repository of Source code for experiment PWAs and Python scripts used to collect and analyse retrieved data https://github.com/xwilen22/thesis-blazor-pwa-evaluation

64