Linköping University | Department of Computer and Information Science Bachelor’s thesis, 16 ECTS | Computer Science 2020 | LIU-IDA/LITH-EX-G--20/062--SE Cloud Gaming: A QoE Study of Fast-paced Single-player and Multiplayer Games Molnspelande: En QoE Studie med Fokus på Snabba Single- player och Multiplayer Spel Sebastian Flinck Lindström Markus Wetterberg Supervisor : Niklas Carlsson Examiner : Marcus Bendtsen Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer- ingsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko- pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis- ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker- heten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman- nens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to down- load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/. Sebastian Flinck Lindström © Markus Wetterberg Students in the 5 year Information Technology program complete a semester-long soft- ware development project during their sixth semester (third year). The project is completed in mid-sized groups, and the students implement a mobile application intended to be used in a multi-actor setting, currently a search and rescue scenario. In parallel they study several topics relevant to the technical and ethical considerations in the project. The project culmi- nates by demonstrating a working product and a written report documenting the results of the practical development process including requirements elicitation. During the final stage of the semester, students create small groups and specialise in one topic, resulting in a bache- lor thesis. The current report represents the results obtained during this specialisation work. Hence, the thesis should be viewed as part of a larger body of work required to pass the semester, including the conditions and requirements for a bachelor thesis. Abstract Cloud computing is a way to deliver high-performance services to clients who would not usually be able to handle the computations on their own. They rely on computers in the cloud performing the calculations and therefore ease the load on the client-side. The goal of this thesis is to find what factors affect the players’ experience and how they affect the player. To handle this, we have done a user-based study on cloud gaming. The users get to play a fast-paced single-player game and a fast-paced multiplayer game against each other, while we collect data about their experiences. During the tests, we manipulated the players’ network conditions, and afterward, they answered questions regarding their quality of experience. From the data collected, we can see that the frame age is the most important measure- ment for determining the players’ in-game performance as well as the quality of experi- ence. We are also able to see that from the quality of service measurements manipulated, the latency is the one affecting the player the most. Results from the multiplayer test would indicate that we can equalize the skill differ- ence between the players without affecting the players quality of experience too much. These results are based on the advantage in ping time as well as frame age. From a developers perspective, this thesis emphasized the need to take frame age into account, and to try to manipulate the different parts of the frame age. The goal would be to ultimately lower the frame age and make the gaming experience more enjoyable for the player. Acknowledgments The authors would like to extend their sincere thanks to Associate Professor Niklas Carlsson at Linköping’s University for his guidance and for playing a decisive role in how this thesis turned out. Special thanks should also go to our fellow students Erica Christensen Weistrand, Sophie Ryrberg, Alexandra Goltsis, Carl Hermod Ekblad and Rami Latif for their invaluable insight and help. v Contents Abstract iii Acknowledgments v Contents vi List of Figures viii List of Tables ix 1 Introduction 1 1.1 Aim............................................ 1 1.2 Research Questions . 2 1.3 Delimitations . 2 1.4 Findings . 2 1.5 Structure . 2 2 Background and Related Work 3 2.1 Background . 3 2.2 Related Work . 5 2.3 Comparison . 6 3 Method 7 4 Single-player Results 12 4.1 Scenario-based QoE Analysis . 12 4.2 Correlation Analysis . 15 4.3 Age-based QoE Analysis . 17 4.4 Other QoS-based QoE analysis . 18 4.5 Model-based Parameter Selection . 19 4.6 Accounting for player differences . 21 5 Multiplayer Results 25 5.1 The impact of Winning on Quality of Experience . 29 5.2 Correlation Analysis for the Combined Data . 30 5.3 Model-based Parameter Selection . 33 6 Discussion 35 6.1 Results . 35 6.2 Method . 37 6.3 The Work in a Wider Context . 40 7 Conclusion 41 vi Bibliography 42 8 Appendix 45 8.1 QoS measurements Top 3 . 45 vii List of Figures 2.1 Steam Link Streaming Diagram . 3 3.1 Screenshot Geometry Wars: Retro Evolved . 8 3.2 Frame Age per Frame . 9 3.3 Screenshot Speedrunners . 11 4.1 QoE Metrics Across all Scenarios . 14 4.2 Score Compared to Baselines Across Scenarios . 14 4.3 Correlation Matrix for Pearson’s r . 15 4.4 Correlation Matrix for Kendall’s t ............................ 16 4.5 Correlation Matrix for MIC . 17 4.6 In-game Performance and QoE Depending on Average Frame Age with a bin-size of50ms............................................ 18 4.7 In-game Performance and QoE Depending on Average Frame Age with a bin-size of 100 ms . 19 4.8 The Smoothed Average for In-game Performance and QoE compared to Different QoS.............................................. 20 4.9 Model Occurrences of Predictors . 21 4.10 In-game Performance and QoE for two Players Depending on Average Frame Age with a bin-size of 50 ms . 22 4.11 Individual Model Occurrences of Predictors . 23 5.1 Win Rate Compared to the Advantage in Ping Time for Player 1 . 25 5.2 Win Rate Compared to the Advantage in Average Frame Age for Player 1 . 26 5.3 General opinion as ping changes . 27 5.4 Graphics score as ping changes . 27 5.5 Interactive Score as ping changes . 27 5.6 General opinion as the average frame age changes . 28 5.7 Graphics score as the average frame age changes . 28 5.8 Interactive Score as the average frame age changes . 28 5.9 QoE Depending on Winner . 29 5.10 Correlation Matrix for Pearson’s r . 31 5.11 Correlation Matrix for Kendall’s t ............................ 32 5.12 Correlation Matrix for MIC . 33 5.13 Model Occurrences of Predictors . 34 viii List of Tables 3.1 Data Collected . 8 3.2 Test Scenarios . 10 3.3 The Lag Factor x Depending on the Difference in Score . 11 4.1 Average Performace and QoE for Test Scenarios . 13 4.2 Top 7 Best Pearson Correlations for QoE and Performance . 15 4.3 Top 7 Best Kendall Correlations for QoE and Performance . 16 4.4 Top 7 Best MIC Correlations for QoE and Performance . 17 4.5 Model including categorical predictor . 24 5.1 Models including winning categorical predictor . 30 5.2 Top 7 Best Pearson Correlations for QoE and Performance . 31 5.3 Top 7 Best Kendall Correlations for QoE and Performance . 32 5.4 Top 7 Best MIC Correlations for QoE and Performance . 33 6.1 Percentage of p > 0.05 for different correlation coefficients . 38 6.2 p-values for Different Correlations Rounded to 4 Significant Figures (3 for MIC) . 39 6.3 Cronbach’s a for Different Games . 39 8.1 Top 3 Best Pearson Correlations for General Opinion Score . 45 8.2 Top 3 Best Pearson Correlations for Graphical Score . 45 8.3 Top 3 Best Pearson Correlations for Interactive Score . 45 8.4 Top 3 Best Pearson Correlations for In-game Performance relative to the Baseline . 45 8.5 Top 3 Best Kendall Correlations for General Opinion Score . 46 8.6 Top 3 Best Kendall Correlations for Graphical Score . 46 8.7 Top 3 Best Kendall Correlations for Interactive Score .