Automated testing for multiplayer Game AI in Sea of Thieves
Robert Masella Rare - Microsoft Studios About Me
• Senior Gameplay Engineer
• Worked on: • Banjo Kazooie: Nuts and Bolts • Kinect Sports games • Sea of Thieves
• Two years on AI Rare
• Microsoft first party studio
• 30+ years of games Sea of Thieves Sea of Thieves What is Sea of Thieves?
• ‘Shared World Adventure Game’
• Two main AI threats: • Skeletons on islands • Sharks in the sea
• Other AI types being worked on Sea of Thieves Tech
• Uses Unreal Engine 4
• Dedicated servers • AI processing runs on server
• AI uses UE4 behaviour trees • With many extended classes This Talk
• How we shipped weekly
• AI behaving correctly on every build
• Using automated testing and continuous delivery Why use automated tests?
• Game as a service means constant build confidence necessary
• Reduce manual testing
• Reduce crunch! Testing AI
• Automated testing not widely used in game development
• AI Unique challenges for testing AI
• Multiplayer Imagine this scenario
• Designer changes a number in AI perception asset
• Causes AI to forget players leaving Line of Sight correctly
• Not spotted by manual test
• Bug released to players Consequences
• Weeks later…
• Players start to notice in build
• More engineer time to track down
• More testing time to verify fix
• Every chance issue could reoccur How to avoid
• Add an automated test
• Run it regularly
• Now bug would be caught before release • Ideally before it enters build Testing at Rare
• Require testing with every check in
• Sea of Thieves now has 10,000 tests
• Test run remotely using TeamCity Automated Testing Organisation
• Run core set of tests before every check in • Run longer running tests periodically
• Run all tests multiple times: • Different platforms • Different configurations (e.g. shipping, debug) • Different conditions (e.g. high latency, high packet variance) Release Process
• Fix gets to players in 30 hrs:
0 hrs 0 min Fix implementation completed 0 hrs 30 min Fix verified on our build farm 0 hrs 30 min Fix submitted 3 hrs 0 min Build built, deployed to test lab and testing begins 5 hrs 0 min Entire build verified 29hrs 0 min Build passed through certification, ready to be published to retail. 30hrs 0 min Build published to retail and available to players Test Types
• Unit
• Integration
• Behaviour Tree
• Multiplayer Integration Test Types
• Favour unit tests over integration tests • 90% of tests are unit tests
• Unit tests don’t cover interaction between big systems • And rarely test production assets
• Ideal to have coverage by all test types Testing Standards
• Names follow Given_When_Then pattern • E.g. AIWithHearingSense_ReceivesNoiseEvent_AIAcknowledgesNoise
• Each test tests one thing
• Make sure you’ve seen the test fail as well as pass
• Minimise boilerplate Running Tests
• Run tests in Unreal Editor Automation window Unit Tests
• Run without creating or ticking world
• Unit tests can have test fixture for boilerplate code
#define IMPLEMENT_AIENTITY_TEST( TestName ) IMPLEMENT_UNIT_TEST( TestName, "AIEntity", AIEntityTestFixture )
IMPLEMENT_AIENTITY_TEST( UpdateTarget_OneEnemyEntitySeen_TargetSetToEntity ) { auto* AIEntity = SpawnTestAIEntity();
auto* EnemyEntity = SpawnEnemyEntity(); AIEntity->AddSeenEntity( EnemyEntity );
AIEntity->UpdateTarget();
TestEqual( AIEntity->GetTargetEntity(), EnemyEntity ); } Integration Tests
• Mostly used Unreal Blueprint system:
• https://docs.unrealengine.com/latest/INT/Engine/Blueprints/ Integration Tests
• Levels with limited game scenario
• Most look for success criteria, or time out and fail Line of Sight integration test
• SkeletonAI_WhenLosesLineOfSightToTarget_MovesToPositionToRegainLineOfSight Line of Sight integration test Line of Sight integration test
• Failing version Line of Sight integration test Line of Sight integration test
• Failing version • Assertion: Test Timed Out Line of Sight integration test
• Passing version Line of Sight integration test Line of Sight integration test
• Passing version Behaviour Tree Testing
• In Unreal, behaviour trees are assets • Tested with integration tests.
• Also required testing for new node types • Preferred testing with unit tests Example Behaviour Tree Node
• Node that triggers an input Behaviour Tree Node Testing
• Add node to a minimal behaviour tree created in code • Created environment required for coded behaviour tree in fixture • Also created helper functions for adding nodes virtual void OnBeforeTest() override { BehaviorTree = CreateTreeRootWithSequence(); }
UBTTask_TriggerInput* CreateTriggerInputTaskNodeAttachedToNode( UBTCompositeNode* ParentNode, UNotificationInputId NotificationId ) { auto* TestTask = NewObject< UBTTask_TriggerInput >(); TestTask->NotificationId = NotificationId; ParentNode->AddChild( TestTask );
return TestTask; } Node Testing
• Test that checks that input is triggered and of correct type
IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( TriggerInputNode_NodeIsRun_CallsCorrectInputNotification ) { bool TestNotificationInput1Called = false;
TestEntity->HandleNotificationCallback = [&]( UNotificationInputId NotificationInputId ) { if ( NotificationInputId == UTestNotificationInputId ) { TestNotificationInput1Called = true; } };
NotificationNode = CreateTriggerInputTaskNodeAttachedToNode( RootNode, UTestNotificationInputId );
RunTreeWithInitialTick(); TestTrue( TestNotificationInput1Called ); } Node Testing
• Test that checks that input is triggered and of correct type Example Behaviour Tree Node II
• Decorator node that compares entity’s health Node Testing II
• Test that checks that decorator fails if health is lower than expected
IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( CompareCurrentHealth_CompareGreaterThanValueHealthIsLess_DecoratorNodeFails ) { bool TaskWithDecoratorExecuted = false; bool LowerPriorityTaskExecuted = false;
auto* Decorator = CreateCompareCurrentHealthDecorator( EFloatValueComparisonType::GreaterThan, 0.1f ); AddTestExecutionChildNodeWithDecorator( RootNode, [&]() { TaskWithDecoratorExecuted = true; }, Decorator ); AddTestExecutionChildNode( RootNode, [&]() { LowerPriorityTaskExecuted = true; } );
AIEntity->SetHealth( 0.05f );
RunTreeWithInitialTick();
TestFalse( TaskWithDecoratorExecuted ); TestTrue( LowerPriorityTaskExecuted ); } Node Testing II
• Test that checks that decorator fails if health is lower than expected Multiplayer Integration Testing
• Most AI integration tests require server only
• Sometimes AI tests require client
• Integration test framework supports tests over network Multiplayer Integration Testing
• Use Yield Nodes to pass execution between server and clients ‘Walking Dead’ Multiplayer Integration Test
• SkeletonAI_Dies_DoesNotMoveDuringDeathOnClient_MP1 ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test
• Failing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test
• Failing version • Assertion failed: ‘check velocity is low in dead state’ ‘Walking Dead’ Multiplayer Integration Test
• Passing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test
• Passing version Testing drawbacks
• Time to create tests
• Tests on an evolving game
• Intermittent integration tests Testing benefits
• Shipped over 100 times in 2 years
• Very low bug count
• Improved code
• Manual testers focus on other things
• No crunch for 4 years! Conclusion - AI In Action
• Now lets see it altogether AI In Action Rare Are Hiring
www.rare.co.uk/careers
Particularly looking for gameplay and AI engineers! Questions?