Automated testing for multiplayer Game AI in Sea of Thieves Robert Masella Rare - Microsoft Studios About Me • Senior Gameplay Engineer • Worked on: • Banjo Kazooie: Nuts and Bolts • Kinect Sports games • Sea of Thieves • Two years on AI Rare • Microsoft first party studio • 30+ years of games Sea of Thieves Sea of Thieves What is Sea of Thieves? • ‘Shared World Adventure Game’ • Two main AI threats: • Skeletons on islands • Sharks in the sea • Other AI types being worked on Sea of Thieves Tech • Uses Unreal Engine 4 • Dedicated servers • AI processing runs on server • AI uses UE4 behaviour trees • With many extended classes This Talk • How we shipped weekly • AI behaving correctly on every build • Using automated testing and continuous delivery Why use automated tests? • Game as a service means constant build confidence necessary • Reduce manual testing • Reduce crunch! Testing AI • Automated testing not widely used in game development • AI Unique challenges for testing AI • Multiplayer Imagine this scenario • Designer changes a number in AI perception asset • Causes AI to forget players leaving Line of Sight correctly • Not spotted by manual test • Bug released to players Consequences • Weeks later… • Players start to notice in build • More engineer time to track down • More testing time to verify fix • Every chance issue could reoccur How to avoid • Add an automated test • Run it regularly • Now bug would be caught before release • Ideally before it enters build Testing at Rare • Require testing with every check in • Sea of Thieves now has 10,000 tests • Test run remotely using TeamCity Automated Testing Organisation • Run core set of tests before every check in • Run longer running tests periodically • Run all tests multiple times: • Different platforms • Different configurations (e.g. shipping, debug) • Different conditions (e.g. high latency, high packet variance) Release Process • Fix gets to players in 30 hrs: 0 hrs 0 min Fix implementation completed 0 hrs 30 min Fix verified on our build farm 0 hrs 30 min Fix submitted 3 hrs 0 min Build built, deployed to test lab and testing begins 5 hrs 0 min Entire build verified 29hrs 0 min Build passed through certification, ready to be published to retail. 30hrs 0 min Build published to retail and available to players Test Types • Unit • Integration • Behaviour Tree • Multiplayer Integration Test Types • Favour unit tests over integration tests • 90% of tests are unit tests • Unit tests don’t cover interaction between big systems • And rarely test production assets • Ideal to have coverage by all test types Testing Standards • Names follow Given_When_Then pattern • E.g. AIWithHearingSense_ReceivesNoiseEvent_AIAcknowledgesNoise • Each test tests one thing • Make sure you’ve seen the test fail as well as pass • Minimise boilerplate Running Tests • Run tests in Unreal Editor Automation window Unit Tests • Run without creating or ticking world • Unit tests can have test fixture for boilerplate code #define IMPLEMENT_AIENTITY_TEST( TestName ) IMPLEMENT_UNIT_TEST( TestName, "AIEntity", AIEntityTestFixture ) IMPLEMENT_AIENTITY_TEST( UpdateTarget_OneEnemyEntitySeen_TargetSetToEntity ) { auto* AIEntity = SpawnTestAIEntity(); auto* EnemyEntity = SpawnEnemyEntity(); AIEntity->AddSeenEntity( EnemyEntity ); AIEntity->UpdateTarget(); TestEqual( AIEntity->GetTargetEntity(), EnemyEntity ); } Integration Tests • Mostly used Unreal Blueprint system: • https://docs.unrealengine.com/latest/INT/Engine/Blueprints/ Integration Tests • Levels with limited game scenario • Most look for success criteria, or time out and fail Line of Sight integration test • SkeletonAI_WhenLosesLineOfSightToTarget_MovesToPositionToRegainLineOfSight Line of Sight integration test Line of Sight integration test • Failing version Line of Sight integration test Line of Sight integration test • Failing version • Assertion: Test Timed Out Line of Sight integration test • Passing version Line of Sight integration test Line of Sight integration test • Passing version Behaviour Tree Testing • In Unreal, behaviour trees are assets • Tested with integration tests. • Also required testing for new node types • Preferred testing with unit tests Example Behaviour Tree Node • Node that triggers an input Behaviour Tree Node Testing • Add node to a minimal behaviour tree created in code • Created environment required for coded behaviour tree in fixture • Also created helper functions for adding nodes virtual void OnBeforeTest() override { BehaviorTree = CreateTreeRootWithSequence(); } UBTTask_TriggerInput* CreateTriggerInputTaskNodeAttachedToNode( UBTCompositeNode* ParentNode, UNotificationInputId NotificationId ) { auto* TestTask = NewObject< UBTTask_TriggerInput >(); TestTask->NotificationId = NotificationId; ParentNode->AddChild( TestTask ); return TestTask; } Node Testing • Test that checks that input is triggered and of correct type IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( TriggerInputNode_NodeIsRun_CallsCorrectInputNotification ) { bool TestNotificationInput1Called = false; TestEntity->HandleNotificationCallback = [&]( UNotificationInputId NotificationInputId ) { if ( NotificationInputId == UTestNotificationInputId ) { TestNotificationInput1Called = true; } }; NotificationNode = CreateTriggerInputTaskNodeAttachedToNode( RootNode, UTestNotificationInputId ); RunTreeWithInitialTick(); TestTrue( TestNotificationInput1Called ); } Node Testing • Test that checks that input is triggered and of correct type Example Behaviour Tree Node II • Decorator node that compares entity’s health Node Testing II • Test that checks that decorator fails if health is lower than expected IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( CompareCurrentHealth_CompareGreaterThanValueHealthIsLess_DecoratorNodeFails ) { bool TaskWithDecoratorExecuted = false; bool LowerPriorityTaskExecuted = false; auto* Decorator = CreateCompareCurrentHealthDecorator( EFloatValueComparisonType::GreaterThan, 0.1f ); AddTestExecutionChildNodeWithDecorator( RootNode, [&]() { TaskWithDecoratorExecuted = true; }, Decorator ); AddTestExecutionChildNode( RootNode, [&]() { LowerPriorityTaskExecuted = true; } ); AIEntity->SetHealth( 0.05f ); RunTreeWithInitialTick(); TestFalse( TaskWithDecoratorExecuted ); TestTrue( LowerPriorityTaskExecuted ); } Node Testing II • Test that checks that decorator fails if health is lower than expected Multiplayer Integration Testing • Most AI integration tests require server only • Sometimes AI tests require client • Integration test framework supports tests over network Multiplayer Integration Testing • Use Yield Nodes to pass execution between server and clients ‘Walking Dead’ Multiplayer Integration Test • SkeletonAI_Dies_DoesNotMoveDuringDeathOnClient_MP1 ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Failing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Failing version • Assertion failed: ‘check velocity is low in dead state’ ‘Walking Dead’ Multiplayer Integration Test • Passing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test • Passing version Testing drawbacks • Time to create tests • Tests on an evolving game • Intermittent integration tests Testing benefits • Shipped over 100 times in 2 years • Very low bug count • Improved code • Manual testers focus on other things • No crunch for 4 years! Conclusion - AI In Action • Now lets see it altogether AI In Action Rare Are Hiring www.rare.co.uk/careers Particularly looking for gameplay and AI engineers! Questions?.