FOCUS MULTIPARADIGM PROGRAMMING MULTIPARADIGM FOCUS PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: ENGINEERING FUN person shooter gameplay. IEEE Soft- Repairing Games ware magazine is not responsible for content placed on unaffiliated websites.) These situations are created by emergence, the result of multiple sim- at Runtime or, How ple interactions operating together to create a complex set of behaviors.1 Ac- cording to Katie Salen and Eric Zim- We Learned to merman, emergence is the key require- ment for games to remain continually engaging, as players will return to a game as long as it provides “variety, Stop Worrying and 2 novelty, and surprise.” Think of Tic- Tac-Toe. For a six-year-old, Tic-Tac- Toe is an exciting game of strategy and Love Emergence intrigue. Who will win when two great, young minds collide? Of course, as we grow older, we realize that playing Tic- Chris Lewis and Jim Whitehead, University of California, Santa Cruz Tac-Toe with any reasonable competi- tor will result in a draw. The game can no longer surprise, and we abandon it for more varied pursuits, such as chess // By repairing games as they execute, Mayet lets developers or Go. In software engineering, emer- focus on creating excellent gaming experiences without gence is often considered an undesir- able confluence of complex compo- worrying about edge cases and untraceable bugs. // nents, indicating an overengineered or poorly understood system, but in video games, emergence is king. Emergence presents a significant ver- ification obstacle: If software cannot be predicted, how can it be verified? Most automated testing techniques only function when the developer knows both the possible inputs and their cor- responding outputs. However, those outputs are, by definition, unknow- CAST YOUR MIND back to some of justcausejump). However, for all the able (if we did know them for all pos- your greatest gaming moments (or good times, there are also the bad: sibilities, we wouldn’t have an emergent ask your gaming friends for theirs). maybe the time you fell through the floor system). Furthermore, even if we could Maybe they include jumping a quad (http://bit.ly/crysissink) or when your begin the onerous task of enumerating bike into an airplane (http://bit.ly/gta- AI buddy’s pathfinding sent him in the possible inputs and outputs, emergence jump). Perhaps it was the simple joy opposite direction (http://bit.ly/fallout- can only be achieved for an adult mind of being killed in an online game by a bugs). How about when you couldn’t do through state space explosion, and our traffic cone (http://bit.ly/halo3cone). any damage at all to opponents in an on- ability to achieve test coverage across Or maybe it was launching yourself line match (http://bit.ly/gtaonlinebug)? those states decays exponentially. Cur- into the air by riding an exploding gas (Note: some of the games mentioned rent automated techniques don’t work, canister, then hijacking a helicopter and referenced in this article feature and the video game industry has re- on the way back down (http://bit.ly/ violent content in the context of first- signed itself to hiring human testers, 0740-7459/11/$26.00 © 2011 IEEE SEPTEMBER/OCTOBER 2011 | IEEE SOFTWARE 53 FOCUS MULTIPARADIGM PROGRAMMING MULTIPARADIGM FOCUS PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: MULTIPARADIGM PROGRAMMING FOCUS: ENGINEERING FUN sometimes numbering in the hundreds, in non-safety-critical software such has come under alien attack, with ex- to manually test games.3 as games, failures aren’t necessarily a plosions going off all around. A failure Complete verification of emergent problem if they don’t cause significant causes the player to glitch through the software is not yet possible. With this harm. For example, if a nonplayer char- floor to the level below, which is bad in mind, we realized that we had to acter (NPC) isn’t spawned in a game enough. Worse still, as the player at- turn the problem of testing games on world, this is indeed a deviation from tempts to go back to the starting point, its head: If we can’t know or enumer- the expected state and a failure has oc- he or she finds that a locker has fallen ate the extent of all possible outputs at curred. However, if the NPC has no into a doorway. From the video, it’s un- test time, what if we instead use the ac- critical interactions in the game, the clear why the failure occurred. Regard- tual output of the system at runtime? failure’s impact is negligible. In this ar- less, the player can’t return to the top With that output, we can then repair ticle, we talk about acceptable failures, of the aircraft carrier, and must reload a saved game. The astute reader might well com- Failures can occur in complex systems ment that the failure was the player falling through the floor, and the locker even if there are no faults in the issue is simply part of a failure cascade. individual software components. Although this assessment is indeed cor- rect, it provides cold comfort for our poor player, who must deal with the situation presented, regardless of how the system if it displays emergent mis- when a failure has occurred but game- it occurred. The player’s inability to behavior, creating a system that can play isn’t significantly impacted, and move through a necessary doorway recover from failures.4,5 To do this, we unacceptable failures, where the failure presents an unacceptable failure, in created Mayet, a system for monitoring causes a strongly negative experience. which the game has essentially reached software at runtime. Mayet evaluates We adopt the term “system under an end state from which the player the running software’s state for invalid test,” which is used often in software can no longer progress. Any number states. If it detects an erroneous state, verification. SUT refers to whatever sys- of situations could lead to a particular it instructs the software to repair itself. tem we’re testing; in this article it will be failure. In this case, the physics engine a Mario-style game. Using SUT avoids might have created an explosion that The Basics of Bugs confusion when describing Mayet, as moved the locker to that position, the When we talk about “bugs,” what do often words like “system” and “pro- designer might have placed the locker we actually mean? Algirdas Avižienis gram” can get conflated together and no to prevent backtracking, a bug in the and his colleagues define bugs at dif- one is really sure what they refer to any- implementation might have spawned ferent points.5 A fault causes an error. more (authors included). the locker by accident, the locker might Faults can be caused by several things, Now armed with our knowledge of have been incorrectly flagged as im- such as human mistakes when coding bug definitions, we can investigate an movable—the list goes on. Instead or vulnerabilities that let external faults actual problem. of looking at causes, we focus on the harm the system. An error is the system problem: the player can’t progress be- state that might lead to a failure. Some Diving In: cause the critical path is blocked. Two errors are handled and corrected in the The Crysis Problem simple solutions present themselves: re- system (such as packet loss in the TCP/ Consider a single instance of the prob- move the locker from the game world, IP protocol), but some can reach the lems caused by emergence. While or teleport the player back to where we system’s external state, in which case browsing YouTube for video game fail- expect him or her to be. Detailing the a failure occurs. A failure occurs when ures (see the sidebar), one of particu- problem and possible repairs is far eas- errors cause a deviation from the sys- lar interest caught our eye. The video ier in this situation than trying to find tem’s expected behavior. is of a playthrough of Crysis, a popu- the causes. In fact, we hypothesize that Failures can occur in complex sys- lar first-person shooter game where the most problems in emergent software tems even if there are no faults in the player takes the role of protagonist, have many more possible causes than individual software components (in shooting enemies while trying to reach solutions. many domains, integration testing will the end of the level. The player is on The Crysis problem shows us two expose these problems). Additionally, the runway of an aircraft carrier that things: 54 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE • Problems can arise from compo- to predict what failures could occur is and rule engine communicate via mes- nent orchestration, not just com- roughly analogous to that required to sages, any refactoring of the SUT ponents themselves. Jonathan Blow create good unit tests that find likely won’t require altering the rules, pro- describes this as the cross-cutting problems at a function level. These vided the SUT conforms to the same concerns problem, noting that rules specify safety properties, which specification. bringing algorithms into a “harmo- express that something bad doesn’t Finally, SUT and rule engine exe- nious whole” is fraught with diffi- happen (such as getting stuck in a wall). cution are kept separate to ensure ex- culty and no existing programming We can’t specify liveness properties, ecution independence. Although rule paradigms address this problem.6 which are properties that say something engines are fast, they aren’t without • Although trying to find the cause good must eventually happen.