Distributed Virtual Environment Scalability and Security John L. Miller September 2011 © 2011 John L. Miller Doctoral thesis for the University of Cambridge Computer Laboratory College: Hughes Hall Abstract Distributed virtual environments (DVEs) have been an active area of research and engineering for more than 20 years. The most widely deployed DVEs are network games such as Quake, Halo, and World of Warcraft (WoW), with millions of users and billions of dollars in annual revenue. Deployed DVEs remain expensive centralized implementations despite significant research outlining ways to distribute DVE workloads. This dissertation shows previous DVE research evaluations are inconsistent with deployed DVE needs. Assumptions about avatar movement and proximity - fundamental scale factors - do not match WoW’s workload, and likely the workload of other deployed DVEs. Alternate workload models are explored and preliminary conclusions presented. Using realistic workloads it is shown that a fully decentralized DVE cannot be deployed to today’s consumers, regardless of its overhead. Residential broadband speeds are improving, and this limitation will eventually disappear. When it does, appropriate security mechanisms will be a fundamental requirement for technology adoption. A trusted auditing system (“Carbon”) is presented which has good security, scalability, and resource characteristics for decentralized DVEs. When performing exhaustive auditing, Carbon adds 27% network overhead to a decentralized DVE with a WoW-like workload. This resource consumption can be reduced significantly, depending upon the DVE’s risk tolerance. Finally, the Pairwise Random Protocol (PRP) is described. PRP enables adversaries to fairly resolve probabilistic activities, an ability missing from most decentralized DVE security proposals. Thus, this dissertations contribution is to address two of the obstacles for deploying research on decentralized DVE architectures. First, lack of evidence that research results apply to existing DVEs. Second, the lack of security systems combining appropriate security guarantees with acceptable overhead. 3 Acknowledgements There are several people without whom this research never could have taken place. Without them, I never would have tried, let alone succeeded. First and foremost, thank you to Mitch Goldberg, Andrew Herbert, and Microsoft Research Cambridge for their support in undertaking a PhD. Andrew provided words of encouragement at the right time to tip the balance towards pursuing this degree. Mitch’s unflagging confidence, optimism, encouragement and support helped make sure I did not abandon the effort when the going got tough. Thank you Mitch! Thank you to my advisor Jon Crowcroft for his guidance and patience with ideas good and bad, and invaluable insights into what constitutes useful research. And thanks to Steve Hand for his feedback on both my thesis proposal and this dissertation, which improved the quality of both. Finally, THANK YOU to my wife Salwa and son Ryan for their patience and support. In addition to putting up with years of my burning the candle both ends, Salwa provided invaluable assistance, ensuring results were correctly presented and human-readable. In addition to her computer science insights and technical proof-reading skills, she is a veteran hand at World of Warcraft, and helped to gather much of the data this research is based upon. Thank you Salwa, and thank you Ryan! 5 6 Table of contents 1 Introduction ....................................................................................................................... 11 1.1 Research statement ............................................................................................................. 11 1.2 Research contributions ........................................................................................................ 12 1.3 Publications .......................................................................................................................... 12 2 General background ........................................................................................................... 15 2.1 DVE responsiveness and correctness .................................................................................. 18 2.2 DVE scalability ...................................................................................................................... 21 2.2.1 Traffic reduction .............................................................................................................. 21 2.2.2 Redistributing simulation workload ................................................................................ 23 2.3 DVE security ......................................................................................................................... 26 3 Towards a realistic DVE workload ....................................................................................... 31 3.1 Introduction ......................................................................................................................... 31 3.2 Relevant background ........................................................................................................... 32 3.2.1 World of Warcraft ........................................................................................................... 32 3.2.2 The Arathi Basin battleground ........................................................................................ 33 3.2.3 Avatar behavior and traffic classification ........................................................................ 36 3.3 Methodology........................................................................................................................ 37 3.4 Analysis ................................................................................................................................ 39 3.4.1 Avatar participation characteristics ................................................................................ 40 3.4.2 Waypoints ........................................................................................................................ 41 3.4.3 Hotspots .......................................................................................................................... 45 3.4.4 Group movement ............................................................................................................ 48 3.5 Conclusions .......................................................................................................................... 52 4 Near-term infeasibility of P2P DVEs ..................................................................................... 55 4.1 Introduction ......................................................................................................................... 55 4.2 Background .......................................................................................................................... 55 4.2.1 World of Warcraft ........................................................................................................... 56 4.2.2 Broadband speeds ........................................................................................................... 56 7 TABLE OF CONTENTS 4.3 Methodology ........................................................................................................................ 58 4.3.1 World of Warcraft network attributes ............................................................................ 59 4.3.2 Simulation trace generation ............................................................................................ 60 4.3.3 Simulator ......................................................................................................................... 62 4.3.4 Topology choices and metrics ......................................................................................... 63 4.3.5 Simulation characteristics ................................................................................................ 64 4.3.6 Simulator validation ......................................................................................................... 65 4.4 Results .................................................................................................................................. 65 4.4.1 Simulation results ............................................................................................................ 66 4.5 Conclusions .......................................................................................................................... 71 5 Trusted auditing of decentralized DVEs ................................................................................ 73 5.1 Related work ........................................................................................................................ 74 5.1.1 DVE threat models ........................................................................................................... 74 5.1.2 P2P DVE frameworks ....................................................................................................... 74 5.1.3 DVE security work ............................................................................................................ 75 5.1.4 General DVE characteristics ............................................................................................. 77 5.2 Threat model .......................................................................................................................