Linköping University | Department of Computer Science Master thesis, 30 ECTS | Datateknik 2020 | LIU-IDA/LITH-EX-A--20/058--SE A Comparison of WebVR and Native VR – Impacts on Performance and User Experience Matteus Hemström Anton Forsberg Supervisor : Vengatanathan Krishnamoorthi Examiner : Niklas Carlsson Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin- istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam- manhang som är kränkande för upphovsmannenslitterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circum- stances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the con- sent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping Uni- versity Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/. Matteus Hemström © Anton Forsberg Abstract The Virtual Reality (VR) market has grown considerably in recent years. It is a technol- ogy that requires high performance in order to provide a good sense of presence to users. Taking VR to the web would open up a lot of possibilities but also poses many challenges. This thesis aims to find out whether VR on the web is a real possibility by exploring the current state of performance and usability of VR on the web and comparing it to native VR. The thesis also aims to provide a basis for discussions on the future of VR on the web. Two identical VR applications were built to make the comparison, one built for the web and one built for native Android. Using these applications, a two-part study was con- ducted with one part focusing on performance and the other on the user experience. The performance evaluation measured and compared performance for the two applications, and the user study used two separate questionnaires to measure the users experienced presence and VR sickness. The performance study shows that the web application is clearly lagging behind the native application in terms of pure performance numbers, as was expected. On the other hand, the user study shows very similar results between the two applications, contradict- ing what would be expected based on performance. This hints at successful mitigation techniques in both the hardware and software used. The study also suggests some interesting further research, such as investigating the relationships between performance, VR sickness, and presence. Other possible further research would be to investigate the effect of prefetching and adaptive streaming of re- sources, and how it could impact VR on the web. Acknowledgments We would like to express our greatest thanks to Niklas Carlsson. Without your help, motiva- tion and continued interest through these years, this thesis would never be finished. A big thank you to Johanna and Mimmi for being our greatest supporters and always believing in us. iv Contents Abstract iii Acknowledgments iv Contents v List of Figures vii List of Tables 1 1 Introduction 2 1.1 Motivation . 2 1.2 Aim............................................ 2 1.3 Research questions . 2 1.4 Delimitations . 3 2 Background 4 2.1 Valtech . 4 2.2 Valtech Store . 4 3 Theory 5 3.1 What is Virtual Reality? . 5 3.2 History of VR . 5 3.3 Immersion . 7 3.4 Presence and Telepresence . 7 3.5 Native VR . 10 3.6 VR on the Web . 10 3.7 Web versus Native . 16 3.8 The Importance of Performance . 16 3.9 Real-time 3D . 19 3.10 Prefetching and Level of Detail . 21 4 Method 23 4.1 Research Method - A Multi-method Strategy . 23 4.2 Virtual Experience Implementations . 25 4.3 Hardware - Test Devices and HMD . 29 4.4 Performance Measurements . 29 4.5 Presence and Simulator Sickness Measurements . 30 4.6 Analysis Procedures . 31 5 VR Applications 32 6 Experimental Results 35 6.1 Performance Test Procedure . 35 v 6.2 Performance Test Results . 35 6.3 SSQ Test Results . 38 6.4 IPQ Test Results . 40 6.5 Correlation between SSQ and IPQ . 41 7 Discussion 42 7.1 Results . 42 7.2 Method . 45 7.3 The work in a wider context . 45 8 Conclusion 47 8.1 Future Work . 47 Bibliography 49 A User Study Test Protocol 56 B Performance Test Protocol 57 C Questionnaires 58 C.1 Demograhics Questionnaire . 58 C.2 Post-session Questionnaire . 59 D IPQ results 64 E SSQ results 65 F WebVR Specification 66 List of Figures 2.1 Valtech Store . 4 3.1 WebVR Overview . 13 3.2 WebGL Overview . 15 4.1 Implementation Overview . 26 4.2 Overview of the 3D Scene Building Process . 27 4.3 Component and Services Architecture . 28 5.1 Start view - WebVR application . 33 5.2 Start view - Native application . 33 5.3 Teleport indicator - WebVR application . 33 5.4 Teleport indicator - Native application . 34 5.5 Information window - WebVR application . 34 5.6 Information window - Native application . 34 5.7 Wall material menu - WebVR application . 34 5.8 Wall material menu - Native application . 34 6.1 Performance Measurements . 36 6.2 Individual FPS Measurements . 37 6.3 SSQ score change between pre and post exposure (calculated as post score ´ pre score)........................................... 39 6.4 SSQ total score change between pre and post exposure without outliers . 39 6.5 IPQ scores for native compared to web (zero means equal scores, positive means native scored higher than web) . 40 vii List of Tables 3.1 Standardized loadings of items on respective subscales . 10 3.2 SSQ categories . 17 6.1 Performance Data Description . 38 6.2 Native correlations . 41 6.3 Web correlations . 41 1 CHAPTER 1 Introduction 1.1 Motivation In recent years, the popularity of VR in the consumer market has grown considerably. There are great challenges with VR technology since it is very sensitive to performance and other factors that may influence usability and presence. The portability and mobility that comes with the combination of smartphones and the web has been proven very successful in the consumer market. Great investments in products such as Google Daydream VR and Samsung Gear VR show that there is a clear strive to take VR to the mobile web. Taking new technology and products to the web is not always easy. History reveals that demanding applications is often built for native platforms to take use of performance and de- velopment tools often provided by the native platform. For example, there is still an ongoing battle with "native versus web" in the business of smartphone applications. With the combination of WebGL and WebVR it is possible to build VR experiences using web technology only, allowing VR to utilize the success and accessability of the web platform. But there is a concern of performance; can real-time rendered VR on the web achieve the high performance requirements? 1.2 Aim The aim of this thesis is to find out if VR is ready for the web and vice versa. The thesis will explore the current state of performance and user experience of real-time rendered VR on the web by comparing with native VR. The result of the thesis could be used as basis when choosing whether or not to use a native platform or the web platform to develop a VR application. It should be noted that there is a rapid development of the technologies used to develop VR and especially for VR on the web, thus the aim of this thesis is also to discuss and predict the future of VR on the web. 1.3 Research questions 1. How does a WebVR application compare to a native VR application in terms of user experience? 2. Can the web platform deliver the performance required for VR applications in terms of frame rate, latency and other performance metrics? 2 1.4. Delimitations 1.4 Delimitations This thesis focuses on the comparison of native VR and WebVR in terms of performance and experienced presence. Although VR is available both as native and web applications on multiple platforms and hardware devices, this thesis will only perform measurements and user study on the Android platform and the GearVR headset due to limited time and access to hardware.