Automated testing for multiplayer Game AI in

Robert Masella - Studios About Me

• Senior Gameplay Engineer

• Worked on: • Banjo Kazooie: Nuts and Bolts • Sports games • Sea of Thieves

• Two years on AI Rare

• Microsoft first party studio

• 30+ years of games Sea of Thieves Sea of Thieves What is Sea of Thieves?

• ‘Shared World Adventure Game’

• Two main AI threats: • Skeletons on islands • Sharks in the sea

• Other AI types being worked on Sea of Thieves Tech

• Uses Unreal Engine 4

• Dedicated servers • AI processing runs on server

• AI uses UE4 behaviour trees • With many extended classes This Talk

• How we shipped weekly

• AI behaving correctly on every build

• Using automated testing and continuous delivery Why use automated tests?

• Game as a service means constant build confidence necessary

• Reduce manual testing

• Reduce crunch! Testing AI

• Automated testing not widely used in game development

• AI Unique challenges for testing AI

• Multiplayer Imagine this scenario

• Designer changes a number in AI perception asset

• Causes AI to forget players leaving Line of Sight correctly

• Not spotted by manual test

• Bug released to players Consequences

• Weeks later…

• Players start to notice in build

• More engineer time to track down

• More testing time to verify fix

• Every chance issue could reoccur How to avoid

• Add an automated test

• Run it regularly

• Now bug would be caught before release • Ideally before it enters build Testing at Rare

• Require testing with every check in

• Sea of Thieves now has 10,000 tests

• Test run remotely using TeamCity Automated Testing Organisation

• Run core set of tests before every check in • Run longer running tests periodically

• Run all tests multiple times: • Different platforms • Different configurations (e.g. shipping, debug) • Different conditions (e.g. high latency, high packet variance) Release Process

• Fix gets to players in 30 hrs:

0 hrs 0 min Fix implementation completed 0 hrs 30 min Fix verified on our build farm 0 hrs 30 min Fix submitted 3 hrs 0 min Build built, deployed to test lab and testing begins 5 hrs 0 min Entire build verified 29hrs 0 min Build passed through certification, ready to be published to retail. 30hrs 0 min Build published to retail and available to players Test Types

• Unit

• Integration

• Behaviour Tree

• Multiplayer Integration Test Types

• Favour unit tests over integration tests • 90% of tests are unit tests

• Unit tests don’t cover interaction between big systems • And rarely test production assets

• Ideal to have coverage by all test types Testing Standards

• Names follow Given_When_Then pattern • E.g. AIWithHearingSense_ReceivesNoiseEvent_AIAcknowledgesNoise

• Each test tests one thing

• Make sure you’ve seen the test fail as well as pass

• Minimise boilerplate Running Tests

• Run tests in Unreal Editor Automation window Unit Tests

• Run without creating or ticking world

• Unit tests can have test fixture for boilerplate code

#define IMPLEMENT_AIENTITY_TEST( TestName ) IMPLEMENT_UNIT_TEST( TestName, "AIEntity", AIEntityTestFixture )

IMPLEMENT_AIENTITY_TEST( UpdateTarget_OneEnemyEntitySeen_TargetSetToEntity ) { auto* AIEntity = SpawnTestAIEntity();

auto* EnemyEntity = SpawnEnemyEntity(); AIEntity->AddSeenEntity( EnemyEntity );

AIEntity->UpdateTarget();

TestEqual( AIEntity->GetTargetEntity(), EnemyEntity ); } Integration Tests

• Mostly used Unreal Blueprint system:

• https://docs.unrealengine.com/latest/INT/Engine/Blueprints/ Integration Tests

• Levels with limited game scenario

• Most look for success criteria, or time out and fail Line of Sight integration test

• SkeletonAI_WhenLosesLineOfSightToTarget_MovesToPositionToRegainLineOfSight Line of Sight integration test Line of Sight integration test

• Failing version Line of Sight integration test Line of Sight integration test

• Failing version • Assertion: Test Timed Out Line of Sight integration test

• Passing version Line of Sight integration test Line of Sight integration test

• Passing version Behaviour Tree Testing

• In Unreal, behaviour trees are assets • Tested with integration tests.

• Also required testing for new node types • Preferred testing with unit tests Example Behaviour Tree Node

• Node that triggers an input Behaviour Tree Node Testing

• Add node to a minimal behaviour tree created in code • Created environment required for coded behaviour tree in fixture • Also created helper functions for adding nodes virtual void OnBeforeTest() override { BehaviorTree = CreateTreeRootWithSequence(); }

UBTTask_TriggerInput* CreateTriggerInputTaskNodeAttachedToNode( UBTCompositeNode* ParentNode, UNotificationInputId NotificationId ) { auto* TestTask = NewObject< UBTTask_TriggerInput >(); TestTask->NotificationId = NotificationId; ParentNode->AddChild( TestTask );

return TestTask; } Node Testing

• Test that checks that input is triggered and of correct type

IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( TriggerInputNode_NodeIsRun_CallsCorrectInputNotification ) { bool TestNotificationInput1Called = false;

TestEntity->HandleNotificationCallback = [&]( UNotificationInputId NotificationInputId ) { if ( NotificationInputId == UTestNotificationInputId ) { TestNotificationInput1Called = true; } };

NotificationNode = CreateTriggerInputTaskNodeAttachedToNode( RootNode, UTestNotificationInputId );

RunTreeWithInitialTick(); TestTrue( TestNotificationInput1Called ); } Node Testing

• Test that checks that input is triggered and of correct type Example Behaviour Tree Node II

• Decorator node that compares entity’s health Node Testing II

• Test that checks that decorator fails if health is lower than expected

IMPLEMENT_UNIT_TEST_BEHAVIOUR_TREE_NODE( CompareCurrentHealth_CompareGreaterThanValueHealthIsLess_DecoratorNodeFails ) { bool TaskWithDecoratorExecuted = false; bool LowerPriorityTaskExecuted = false;

auto* Decorator = CreateCompareCurrentHealthDecorator( EFloatValueComparisonType::GreaterThan, 0.1f ); AddTestExecutionChildNodeWithDecorator( RootNode, [&]() { TaskWithDecoratorExecuted = true; }, Decorator ); AddTestExecutionChildNode( RootNode, [&]() { LowerPriorityTaskExecuted = true; } );

AIEntity->SetHealth( 0.05f );

RunTreeWithInitialTick();

TestFalse( TaskWithDecoratorExecuted ); TestTrue( LowerPriorityTaskExecuted ); } Node Testing II

• Test that checks that decorator fails if health is lower than expected Multiplayer Integration Testing

• Most AI integration tests require server only

• Sometimes AI tests require client

• Integration test framework supports tests over network Multiplayer Integration Testing

• Use Yield Nodes to pass execution between server and clients ‘Walking Dead’ Multiplayer Integration Test

• SkeletonAI_Dies_DoesNotMoveDuringDeathOnClient_MP1 ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

• Failing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

• Failing version • Assertion failed: ‘check velocity is low in dead state’ ‘Walking Dead’ Multiplayer Integration Test

• Passing version ‘Walking Dead’ Multiplayer Integration Test ‘Walking Dead’ Multiplayer Integration Test

• Passing version Testing drawbacks

• Time to create tests

• Tests on an evolving game

• Intermittent integration tests Testing benefits

• Shipped over 100 times in 2 years

• Very low bug count

• Improved code

• Manual testers focus on other things

• No crunch for 4 years! Conclusion - AI In Action

• Now lets see it altogether AI In Action Rare Are Hiring

www.rare.co.uk/careers

Particularly looking for gameplay and AI engineers! Questions?