1500 Archers on a 28.8: Age of Empires – Promo Network Programming in Age of Empires and Beyond J. Smed, T. Kaukoranta and H. Hakonen Gamasutra (First appeared at the Game Developer’s Conference, 2001) http://www.microsoft.com/games/empires/multimedia.htm March 22, 2001 Age of Empires – Real Time Strategy Introduction • Build This page explains Age of Empires (AoE) 1 • Combat • • Explore and 2 multiplayer implementation. • Explains: – Design Architecture – Implementation – Lessons learned • Also for the “future” RTS by Ensemble – (Age of Mythology, AoM) Outline AoE: Multiplayer Design Goals • Introduction (done) • Support for 8 players • Implementation (next) • Smooth simulation over modem, Internet, LAN • Lessons Learned • Target platform: 16 MB P-90, 28.8 modem Improvements for AoE 2 • – (AoM is PX-450, 56.6 modem) • RTS3 – At 15 frames per second (one frame every • Summary 67 ms) • Use (existing) Genie engine – next 1 AoE in Early Stages (1 of 2) AoE in Early Stages (2 of 2) • Game engine was running • Wanted: army on army, large supporting structure, – 2d, single-threaded game loop … (“1500 archers on a …”) – Sprites in 256 colors • Time to complete each simulation step varied: – Reasonably stable – Render time changes with number of units • Large map, thousands of objects, trees could be – When scrolling chopped, animals ran around … – AI computation time varied with units or time • Breakdown: – As much as 200 ms (larger than a frame time!) – 30% graphic rendering Bandwidth a critical resource: – 30% AI • – 30% simulation – Passing x,y coordinates, status, action, facing damage … limit of 250 moving units at most • “Compelling” single-player game already – (MLC: 1 bytes each Æ 6 actions x 250 units x 15 – (ie- “Don’t ruin it!”) updates per second ~ 160 Kbps) Communication Turns Simultaneous Simulations • Separate communications turns from frame rendering • Each PC ran exact same simulation • Schedule commands for later time – Allows for some variance in network and turn – Synchronized game time processing – Synchronized random number generators • Turns typically 200 ms in length Still – Send all commands entered that turn, but schedule • them for 2 turns later – Internet latency from 20 to 1000 – Process any scheduled turns milliseconds – Variable time to process each step • Needed a more responsive approach The Need for Speed Control Speed Control • Since all machines in • Each client calculates frame rate “lock step”, can only – Since varies with game state, use moving average run as fast as slowest – Send with “Turn Done” message machine – Use to achieve “minimum” frame rate • Each client measures round-trip “ping” time – Process – Since varies with Internet traffic, use largest for communications, all players (MLC: assume moving average) render turn, send out • fps + rtt Æ 2-bytes total overhead new commands • After getting “Turn Done” messages • “Lag” if – Adjust target frame rate (based on local PC render – One machine slows rate) down and others wait – Adjust communication turn (based on ping-times + – Delayed or lost remote PC render rates) Internet data – Weighted, so only “laggy” during worst spikes • (Examples next) 2 Speed Control Transport Protocol - UDP • Unreliable, so each client handles command ordering, drop detection and re-sending 1) Typical communication turn – “When in doubt, assume it dropped” • Messages arriving from past turns are discarded If out of order message received, request 2) High latency, normal machine • a resend of supposedly “missing” messages – Note, if really out of order, will get duplicate so must account for • If ack is “late”, then assume lost so resend 3) High latency, slow machine Side Problems – Out of Synch Side Benefit - Cheat Prevention “In every project, there is one stubborn bug that goes all the way to the wire…” • Simultaneous simulations means games are – Microsoft product manager identical • Subtle, since small errors multiply • If there is a discrepancy, game stopped – Example – a deer slightly out of alignment, • Prevents cheaters from using hacked client causes villager to “miss” so no meat, causing different food amounts • But there still could be cheating via Checksums (objects, pathing, targeting …), information exposure • but always something – Wade through 50 MB of message traces – (MLC: different game states when commands are lost or are too late?) Lesson: Know Your User Outline • Each genre is different • Introduction (done) – RTS • Implementation (done) – FPS • Lessons Learned (next) – Sports • Improvements for AoE 2 – MMORPG – … • RTS3 1) Know latency expectations Summary • 2) Prototype multiplayer aspects early 3) Watch for behavior that hurts multiplayer performance 3 Know Your User - Expectations Know Your User - Actions • Simulated latency with single-player engine – Look for: good, sluggish, jerky, horrible • Users clicked once per 1.5 – 2 seconds • For AoE • During battle, spikes of 3-4 clicks per – 250 milliseconds (ms) not noticed second – 250-500 ms playable • Many of the commands are repeats – 500+ ms noticeable – Turn is longer than command • Consistent slow (500 ms) better than – Unit takes several turns to process “jerky” (80 ms – 500 ms) • Add “filter” to remove repeat commands – Suggested picking conservative turn length – Make change to new turn length gradually Lesson: Educating the Developers Lesson: Metering is King • Get programmers to think about • Make performance meters human readable multiplayer (distributed systems!) and understood by testers • Multiple, independent processing – Need to educate testers – Command request might happen later (or not – Testers can notice differences, help at all) determine where problems are • Additional calls to random numbers can • Keep running all the time throw off synchronization – Low impact – Random sounds or animations on high-perf – Early development measurements may computers need to save and re-seed to keep change in later game random in-synch Misc Lessons Outline • Verify 3rd party code (AoE used Microsoft’s DirectPlay) • Introduction (done) – Is “guaranteed delivery” guaranteed? Implementation (done) – Does product have hidden bottlenecks? • • Create simulations and stress tests • Lessons Learned (done) – Isolated connection flooding, simultaneous • Improvements for AoE 2 (next) connections, dropping of guaranteed packets… • Test with modems as early as possible • RTS3 – Hard to isolate network problems (ex: could be ISP) • Summary – Helps make sure not the networking part – Although tests harder (and not as fun), do as many modem tests as LAN tests 4 Improvements for Age of Empires 2 Outline – the Age of Kings • New multiplayer features • Introduction (done) – Recorded games • Implementation (done) – File transfer (custom maps) • Lessons Learned (done) – Statistics tracking (on “The Zone”) • Improvements for AoE 2 (done) • Recorded games helps in debugging • RTS3 (next) – Can replay exactly to spot problems – Overview – New features and tools • Summary RTS3 – Beyond AoE RTS3 • Forget DirectPlay Æ created own library • RTS3 is “codename” for next generation – Use in subsequent games Ensemble RTS (probably Age of Mythology) – Integrated with BANG! • Add 3-d capability (used BANG!) • Fully 3-d world meant frame rate maybe an • Multiplayer requirements issue – Again, complex maps, thousands of units, – Overall smoothness from frame rate Internet play impacted – More than 8 players (AoM allows 12) – Also more variation – Still modem, but 56.6k Realized play-testing was iterative, so – May be firewalls and NAT boxes so peer-to- • peer harder wanted multiplayer running earlier An OO Approach Peer-to-Peer Topology • Strengths – Reduces latency – No central point of failure (can continue game if one client leaves) • Weaknesses – More active connections (n) – Impossible to support some NATs (Make protocol specific parts as small as possible) 5 Net.lib (1 of 2) Net.lib (2 of 2) • Level 1: Socks • Level 3: Multiplayer – Fundamental C API, – Client, Session, Channel Berkeley sockets (ordered or unordered), Time • Level 2: Link Sync – Transport layer services • Level 4: Game Communications – Packet, Link, Listener, – RTS functionality (could Data stream, Network define others for different Address, Ping genres) New Tools and Features Summary • Improved synch – Geared towards rapid turn around time • Peer-to-Peer for responsiveness from out-of-synch bugs • Synchronous simulation for scalability – Compile-out extra synch debugging code upon release • Compensation for heterogeneity in clients • Console commands and config and variability in networking – Simple text “hooks” to game engine • Overall – In multiplayer, passed to other clients and – Multiplayer “feels” like single player executed there – Success! – Testing without writing additional code 6.