By Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei HOW SENSITIVE ARE ONLINE GAMERS TO NETWORK QUALITY? Game-playing time is Network quality is critical to real-time, interactive, online game play. Yet even though today’s best-effort strongly related to does not provide quality of service (QoS) network QoS, helping guarantees, the popularity of online gaming is only determine user increasing. This popularity may be because QoS is satisfaction and deliver not really a must-have characteristic or because play- ers are accustomed to struggling with and playing better service quality to through unfavorable network conditions. online gamers. Ironically, despite the popularity of online games, complaints about high “ping times” and “lags” con- tinue to surge in game-player forums (such as www.gamedev.net/community/forums/). It would appear that most players view network latency and loss as a major hindrance to an enjoyable gaming experience. Here, we investigate whether players are really as sensi- tive to network quality as they claim. If they are, we would like to answer whether they quit games earlier than they might otherwise due to unsatisfactory network conditions.

A number of studies have sought to evalu- outweigh their QoS-intolerance and affect ate the effect of network quality on online their decisions. Besides, it is very costly to gamers [1–3, 5, 7–9, 11]. In most of them, a scale up such studies by including more series of games is played in a controlled net- human subjects. Meanwhile, objective evalu- work environment in which the gaming ations are usually based on user performance experience of subjects is graded either subjec- in a specific context (such as the number of tively or objectively. Subjective tests require kills in a shooting game, the time needed to participants to report their feelings about complete each lap in a racing game, and the playing games with different levels of network capital accumulated in a strategy game). QoS. While this approach might capture However, game scores are highly dependent human perceptions in a controlled situation, on player skills, game design, and content, so it cannot measure how players react to poor results are not comparable and generalizable game quality in real life, as other factors may across different games.

34 November 2006/Vol. 49, No. 11 COMMUNICATIONS OF THE ACM E ONLINE GAMERS TO ? Psychologically, the pleasing sensation players expe- UserJoy Technology in (www.ewsoft.com. rience in online games is analogous to being in the tw). MMORPGs (such as EverQuest and World of flow state after taking a mood-changing substance like Warcraft) are networked computer role-playing marijuana. One study [6] found that the most signif- games in which a large number (perhaps 10,000) of icant factors related to the flow experience in online players interact with one another in a virtual world. gaming are skill, challenge, and involvement. The While many game genres are round- or stage-based, game-playing experience can be described as so plea- a design that allows players to regain consciousness surable that gamers are unaware of the time that of the real world, the adventure in role-playing passes while they’re playing the game. According to games is continuous; no explicit mechanism forces the theory laid out in [6], if the feeling of involvement players to take a break. in the virtual world is diminished by network lags, An uninterrupted sense of immersion in a virtual users become more conscious of the real world, miti- environment is one of the main attractions of gating their sense of time distortion. Some players MMORPGs because it entices players into a flow may simply decide to quit a game as soon as they experience; other attractions are elaborate reward detect unacceptable lags. For these reasons, we con- cycles and evolving social networks [10]. The impor- jecture that game-playing time is affected (to some tance of the sense of immersion implies that network extent) by the network quality players experience. quality, a key factor in maintaining a perfect sense of There are certainly exogenous reasons other than immersion, strongly influences the flow experience of the quality of network conditions that might affect MMORPG players; therefore, game-playing time gamers’ decisions to continue with (or leave) a game. should be shorter if the network conditions that users For example, they might leave a game due to their experience are unsatisfactory. schedule constraints or mental state. On the other Another reason for investigating MMORPGs is hand, they may be tied by social bonds, no matter that they are relatively slow-paced compared to other how unstable the game play might be. Although the popular genres (such as first-person shooter) that usu- behavior of individual players is highly variable and ally require players to make split-second decisions. In unpredictable, we show here that, overall, there is a addition to the pace of a game, there is a great deal of consistent and strong link between player departure variety in how players control their virtual characters. decisions and network QoS. In fast-action genres (such as shooters) players must instruct characters “what” actions to take and “how” NETWORK QUALITY VS. GAME-PLAYING TIME to perform them; for example, to move a character to Our conjecture is validated by real-life traces taken a new location, a player must control each step it from the commercial massively multiplayer online needs to take (such as three steps west followed by five role-playing game Shen Zhou Online developed by steps north). On the other hand, in slow-action gen-

COMMUNICATIONS OF THE ACM November 2006/Vol. 49, No. 11 35 res (such as MMORPGs Korean MMORPGs and war strategy games) 4 played in [4], players instruct characters showing that the average 3 only “what” to do; that is, ) session time ranges from r u o h they point out only the ( 80 to 120 minutes. How-

e 2 m i t

location the character ever, the difference in y a l p should move to; it then individual game-playing e 1 m a

automatically heads G times is quite large; for toward it via a route that is 150 200 250 300 example, the shortest either predetermined by a Network latency (msec) 20% of sessions span less game designer or com- 6 than 40 minutes, but the puted on-the-fly. top 20% of players spend

By any standard, ) 4 more than eight continu- r u o h

MMORPGs are slow- ( ous hours in a game.

e m i t action games with less-strin- 2 We use graphical plots y a l p

gent service requirements e to illustrate the difference m a than fast-action games. G 0 in the game-playing time Therefore, they could be 52515 35 45 55 of sessions experiencing viewed as a baseline of real- Network delay variation (msec) different levels of net- time interactive online work quality (see Figure games such that if network 1). The figure depicts the 6 )

QoS frustrates MMORPG r association of game-play- u o h

( 4 players, it should also frus- ing time with network e m i t trate players of other game latency, network delay y a l 2 p

genres. e variation, and network m a

To play Shen Zhou G 0 loss rate, respectively. All Online, thousands of play- 10-10 10-8 10-6 10-4 10-2 three plots indicate that ers pay a monthly sub- Network loss rate the more serious the net- scription fee at a work impairment players convenience store or experienced, the sooner Figure 1. Relationship online. As is typical of between game session time and they were likely to leave the game. The changes in MMORPGs, players can network QoS factors. game-playing time are surprisingly significant. For engage in fights with ran- instance, gamers who experienced 150msec latency dom creatures, train them- played four hours on average, but those experiencing selves to acquire particular skills, partakeChen in fig250msec 1 (11/06) latency played for only one hour on aver- commerce, or take on a quest. With the help of the age—a ratio of 4:1. Moreover, variations in network Shen Zhou Online staff, we recorded all delay and network loss induce more variation in game- inbound/outbound game traffic of the game servers playing time (note the span of the y-axis of the located in Taipai, Taiwan, a total of 15 140-game ses- graphs). sions over two days. Having demonstrated that players are not only sen- We extracted the network performance for each sitive but also reactive to network quality, we still must session based on the sequence number and flags in the define “good quality” and “bad quality.” For example, TCP packet header. We computed network latency given two QoS configurations—“low latency with based on the difference between the acknowledge- moderate variation” and “moderate latency with low ment and the delivery time of nonretransmitted game variation”—which one is better? To answer, we pro- packets and inferred the network loss rate based on pose a model that describes the changes in game-play- the receipt (or lack of receipt) of TCP acknowledge- ing time due to network QoS. This allows us to grade ments. The observed players were spread over 13 the overall quality of game playing based on a set of countries, including , India, , and network performance metrics (such as latency and , and hundreds of autonomous systems, loss) in terms of user satisfaction. manifesting the heterogeneity of network-path char- acteristics—therefore the generality of this trace. On MODELING GAME-PLAYING TIME average, players stayed for 100 minutes after joining a We find that the survival model, which is often used game. This statistic is consistent with the statistics of in the medical field to describe the relationship

36 November 2006/Vol. 49, No. 11 COMMUNICATIONS OF THE ACM FORECASTING WHEN A PLAYER WILL LEAVE A GAME could provide useful hints for system performance optimization and resource allocation. between patients’ survival time and the treatment graph sorts sessions, grouping them by player risks, they receive, provides a good fit for describing the which are defined according to the first equation, to reactions of game players to network quality [3]. The estimate the level of player intolerance to poor net- derived model takes network QoS factors as the work conditions. We observed that, at the macro input and computes the departure rate of online level, the prediction is rather close to the actual time players as the output. The regression equation is observed, suggesting that a service provider can pre- defined as follows: dict how long a given individual player will stay after joining a game and optimize resource allocation departure rate ␣ exp( 1.6 ϫ network latency ϩ accordingly. 9.2 ϫ network delay variation ϩ Please note that while this methodology is applied 0.2 ϫ log(network loss rate)). to all kinds of online games, the exact equation about players’ QoS sensitivity may depend on a specific It illustrates that the player departure rate is game design (such as the transport protocol and

roughly proportional to ) client-prediction tech- r the exponent of the u niques used), varying o h

weighted sum of certain ( among games. 8 Observation e network performance Prediction m i APPLYING PLAYER metrics, where the weights t

y ENSITIVITY a 6 S

reflect the effect of each l p type of network impair- Even though we are e

ment. The coefficients can m unable to change the fact

a 4 g be interpreted by the ratio that the Internet is not e of the departure rates of g QoS-guaranteed, perhaps a r 2 two sessions. For example, e leaving players dissatisfied v suppose two players—A A due to poor network con- and B—join a game at the 0 ditions, we can still same time and experience -1.6 -1.2 -0.8 -0.4 0 0.4 improve matters by similar levels of network Risk exploiting their sensitivity loss and delay variations, to network QoS. except that their network Figure 2. Actual vs. Improving user satisfaction. Given that we can model-predicted game-playing latency is 100msec and time for session groups quantify the risk of players leaving a game due to 200msec, respectively. The sorted by their risks. unsatisfactory QoS, systems can be designed to auto- ratio of their respective matically adapt to network quality in real time in departure rates can then be computed by exp((0.2Chen .fig or2der (11/06) to improve user satisfaction. For example, we where 1.6 is the coefficient of net- might enhance the smoothness of game playing in ,1.2 ف (ϫ 1.6 (0.1 work latency. That is, at every moment during the high-risk sessions by increasing the packet rate (if the game, the probability that A will leave the game is 1.2 risks are caused by long propagation delay or random times greater than the probability that B will leave the loss on a noisy link rather than transient congestion) game. or the degree of data redundancy, so players would be Given the strong relationship between game play- less likely to leave prematurely. Scarce resources, ing time and network QoS, we can “predict” the for- including processing power (for handling player mer if we know the latter. Forecasting when a player requests and computing game states) and network will leave a game could provide useful hints for system bandwidth (for dispatching the latest game states), performance optimization and resource allocation. To could be allocated more appropriately to sessions assess the feasibility of forecasting game playing time, based on session risk scores. Resource allocation could we compared actual time and model-predicted time be deliberately biased toward high-risk sessions char- for Shen Zhou online players (see Figure 2). The acterized by poor network quality, while simultane-

COMMUNICATIONS OF THE ACM November 2006/Vol. 49, No. 11 37 ously maintaining a reasonable level of player satisfac- applications are normalized and compared with one tion in low-risk sessions. another? We leave it as an open question. Only a Optimization of network infrastructure. Based on detailed analysis of the relationship between each the first equation, network quality can be assessed by component in a network system and user perceptions way of a single value, making it possible to compare can deliver a clear answer. c different levels of quality in terms of user satisfaction. For example, according to our scoring, players prefer References a setting of 200msec latency and 0.1% loss rate over 1. Armitage, G. An experimental estimation of latency sensitivity in mul- tiplayer Quake 3. In Proceedings of the 11th IEEE International Confer- a setting of 100msec latency and 1% loss rate, because ence on Networks (Sydney, Australia, Sept. 28–Oct. 1). IEEE Press, packet loss is less tolerable than network latency. As Piscataway, NJ, 2003. 2. Beigbeder, T., Coughlan, R., Lusher, C., Plunkett, J., Agu, E., and different performance metrics usually reflect design Claypool, M. The effects of loss and latency on user performance in trade-offs, this score is useful for optimizing overall Unreal Tournament 2003. In Proceedings of NetGames 2004 (Portland, user satisfaction. For instance, as our model indicates OR, Aug). ACM Press, New York, 2004, 144–151. 3. Chen, K.-T., Huang, P., Wang, G.-S., Huang, C.-Y., and Lei, C.-L. that players are less tolerant of large delay variations On the sensitivity of online game playing time to network QoS. In Pro- than they are of high latency, providing a smoothing ceedings of IEEE INFOCOM 2006 (Barcelona, Spain, Apr. 23–29). IEEE Press, Piscataway, NJ, 2006. buffer at the client side—incurring additional latency 4. 4Gamer.net. Gametrics weekly Korea MMORPG population survey; but smoothing the pace of game play—would www.4gamer.net/specials/gametrics/gametrics.shtml. improve the overall game experience. 5. Henderson, T. and Bhatti, S. Networked games: A QoS-sensitive application for QoS-insensitive users? In Proceedings of the ACM SIG- Network troubleshooting. In order to provide COMM Workshop on Revisiting IP QoS (Karlsruhe, Germany, Aug. continuous high-quality game service, providers must 27). ACM Press, New York, 2003, 141–147. monitor network conditions between servers and cus- 6. Ila, S., Mizerski, D., and Lam, D. Comparing the effect of habit in the online game play of Australian and Indonesian gamers. In Proceedings tomer networks, detecting problems in real time of the Australia and New Zealand Marketing Association Conference before customer complaints flood the customer ser- 2003 (Adelaide, Australia, Dec. 1–3, 2003). 7. Nichols, J. and Claypool, M. The effects of latency on online Madden vice center. However, considering the large number of NFL Football. In Proceedings of Network and Operating System Support online users (hundreds of thousands is not uncom- for Digital Audio and Video 2004 (Kinsale, County Cork, Ireland, June mon for some popular games), it would be prohibi- 16–18). ACM Press, New York, 2004, 146–151. 8. Oliveira, M. and Henderson, T. What online gamers really think of the tively expensive, even impractical, to monitor the Internet? In Proceedings of NetGames 2003 (Redwood City, CA, May status of network paths between all intercommuni- 22–23). ACM Press, New York, 2003, 185–193. 9. Yasui, T., Ishibashi, Y., and Ikedo, T. Influences of network latency cated networks in real time. Instead, game operators and packet loss on consistency in networked racing games. In Proceed- can track user gaming time, which is much more cost- ings of NetGames 2005 (Hawthorne, NY, Oct. 10–11). ACM Press, effective. Since online gamers are sensitive to network New York, 2005, 1–8. 10. Yee, N. The Daedalus Project; www.nickyee.com/daedalus. conditions, a series of unusual (premature) departures 11. Zander, S. and Armitage, G. Empirically measuring the QoS sensitiv- over a short period might indicate abnormal network ity of interactive online game players. In Proceedings of the Australian Telecommunications Networks and Applications Conference 2004 (Bondi conditions and thus automatically trigger appropriate Beach, Sydney, Australia, Dec. 8–10). ATNAC, Australia, 2004, remedial action. 511–518.

CONCLUSION Discussion of QoS provisioning for real-time inter- Kuan-Ta Chen ([email protected]) is an assistant research fellow in the Institute of Information Science at Academia Sinica, active applications, including online gaming, is as , Taiwan. lively as it is because of the popularity of the appli- Polly Huang ([email protected]) is an associate cations and the complexity of the problem. One of professor in the Department of Electrical Engineering and director of the main obstacles to providing a satisfactory user the Network and Systems Laboratory at National Taiwan University, Taipei, Taiwan. experience is that, unlike system-level performance Chin-Laung Lei ([email protected]) is a professor in the metrics (such as bandwidth and latency), user satis- Department of Electrical Engineering and director of the Distributed faction is intangible and unmeasurable. The key to Computing and Network Security Laboratory at National Taiwan addressing this problem is our ability to measure University, Taipei, Taiwan. user opinions about network performance objec- This work is supported in part by the National Science Council under Grants No. tively and efficiently. We have shown that game- NSC 94-2218-E-002-074 and NSC 95-3114-P-001-001-Y02 and by the Taiwan playing time is strongly related to network QoS and Information Security Center, National Science Council, under Grants No. NSC 94- is thus a potential indicator of user satisfaction. Nev- 3114-P-001-001Y and NSC 94-3114-P-011-001. ertheless, user perception is inevitably influenced by the design and implementation of each application. How can the measure of a user’s experience be gen- eralized so the satisfaction measures for different © 2006 ACM 0001-0782/06/1100 $5.00

38 November 2006/Vol. 49, No. 11 COMMUNICATIONS OF THE ACM