Watch Dogs 2 - PC Adaptation Success Story with NVIDIA Jaakko Haapasalo, Producer, GeForce Titles (NVIDIA) Marius Tudorache, Eastern European PC Director (Ubisoft) Farid Rzaev, Developer Technology Engineer (NVIDIA) Oleh Kuznetsov, Senior Programmer (Ubisoft) November 2016… [T]here is a wide range of options that will allow you to cater the experience to your style, which is just what a PC port should offer. The Watch Dogs 2 PC port is great [...] it runs well, has a ton of graphical options, and comes with a complete set of quality- of-life adjustments for mouse The PC version is superb, and keyboard players. and the game is one of the bright spots in this fall’s slate of games. developer.nvidia.com 2 14 out of 14 PC-specific mentions in critic reviews on Metacritic are positive “Very Positive” - 82% on Steam developer.nvidia.com 3 Success PC Platform developer.nvidia.com 4 Agenda Game Ready Quality Program Partnering with Watch Dogs 2 PC – Through The Numbers Ubisoft PC Quality, and working with NVIDIA Technical Case Study: Eliminating Stutter Learnings developer.nvidia.com 5 Game Ready Quality Program Jaakko Haapasalo (NVIDIA) developer.nvidia.com 6 Program Overview 1. PC Technical Requirements Checklist 2. Stability, Performance & Stutter Analysis 3. Minimum & Recommended Specs developer.nvidia.com 7 (see also) Optimal Performance Settings Game Ready Drivers GTM developer.nvidia.com 8 Title Engagement T – 12mo T – 6mo T – 3mo T – 2mo T – 1mo Launch Live Testing & Optimization GTM Review developer.nvidia.com 9 1. TRC (Technical Requirements Checklist) PC platform requirements to validate your game against 3 review milestones Focused on technical quality and readiness, such as resolution 40 requirements support, UI scaling, frame rate, smoothness, settings, and correctness of various effects. 16 titles tested Review involves several play- throughs at different settings, in 2016 pilot significant investment in QA time developer.nvidia.com 10 Requirement Categories •Required (P0) •Core •Recommended (P1) •Quality of Life •Advised (P2) •Forward-looking developer.nvidia.com 11 Requirements and Rating •Description •Additional information (context, intent and failure modes) •Criteria for Full, Partial and Failed compliance •Overall TRC rating as a weighted average (0..100) developer.nvidia.com 12 Timeline T – 12mo T – 6mo T – 3mo T – 2mo T – 1mo Launch Live Title Engagement GTM Review TRC TRC TRC Final Preview Update Results developer.nvidia.com 13 2. Stability, Performance and Stutter Get Build Improve Test Analyze developer.nvidia.com 14 Timeline T – 12mo T – 6mo T – 3mo T – 2mo T – 1mo Launch Live Title Engagement GTM Review TRC TRC TRC Final Preview Update Results Performance & Stutter Analysis developer.nvidia.com 15 3. Minimum & Recommended Specs 60FPS Target MinSpec Benchmark Search “Challenging” MinRec Search (Playthrough!) Benchmark GPU x CPU x SYSMEM RecSpec 1K, 2K, Low / High 4K Presets High: IQ > Any Other Platform developer.nvidia.com 16 Timeline T – 12mo T – 6mo T – 3mo T – 2mo T – 1mo Launch Live Title Engagement GTM Review TRC TRC TRC Final Preview Update Results Min/Rec Spec Min/Rec Spec Min/Rec Min/Rec Prelims Results Check Check Performance & Stutter Analysis developer.nvidia.com 17 WD2 Game Ready: The Numbers Jaakko Haapasalo (NVIDIA) developer.nvidia.com 18 PC Testing TWIMTBP Labs: •7 months •26 builds •10 Stutter & performance tests •5 Min & Rec Spec tests •3 TRC reviews developer.nvidia.com 19 TRC Progression TRC Rating 100 90 80 70 60 50 40 30 20 10 0 1-Jan 3-Jan developer.nvidia.com 20 Min & Recommended Specs: 1K & 2K 65 60 55 Series1 Series2 50 Series3 45 40 1-Jan 2-Jan 3-Jan 4-Jan 5-Jan developer.nvidia.com 21 Partnering with NVIDIA Marius Tudorache (Ubisoft) developer.nvidia.com 22 New NVIDIA-Ubisoft collaboration process •Transparent communication •Smoke-tested builds •Delivery of builds and performance reports •TRC simulations •Post-Mortem and learnings across projects •Dedicated JIRA for each project - tracking bugs outside of e-mails ! Think co-dev instead of tech support ! developer.nvidia.com 23 PC Testing – UBI PC Requirements •13 months (1st check - October 2015) •320+ builds (WD2-GAME-PC2->PC15x + RAW) •~28 performance/TRC tests •~28 Min & Rec Spec tests •7 TRC official reviews (Alpha/Beta/Master/D1P) developer.nvidia.com 24 Ubisoft - Winning PC gamers After 20 years of PC releases: •Knowledge & Information Sharing Management for PC •Internal PC Summits •Improving PC Technical Requirements •Developing PC Communities - internally, externally, E-Sports •Improving communication with PC gamers developer.nvidia.com 25 Current UBI TRC Structure (WD2 proto.) 101 Tech. requirements •Short title – for easy referencing •Intent •Requirement description •Exemption •Terminology •TRC Rating per requirement – different for each milestone •Remarks •Overall Rating – work in progress developer.nvidia.com 26 Using target TRCs as tech design •Clarity •Predictability •Transparency •Continuity •Conformity Major Challenge: Public Tech Requirements (open-sourced to communities) developer.nvidia.com 27 WD2 communicated Min & Reco We wanted to express: •Performance-based specs split per GPU series •Coverage (old to new gen) •Clarity for casual gamers developer.nvidia.com 28 developer.nvidia.com 29 Detecting and Eliminating Stutter Farid Rzaev (NVIDIA) and Oleh Kuznetsov (Ubisoft) developer.nvidia.com 30 Agenda - Stutter overview - Stutter detection - Stutter analysis - Fixing stutter - Results www.gameworks.nvidia.com 31 Stutter? Hitches? Everybody knows what it is…. www.gameworks.nvidia.com 32 10 20 30 40 50 60 70 What What stutter/hitches are? Let’s Define 0 1 - 11 Inconsistency in performance between frames 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 www.gameworks.nvidia.com 191 201 FrameTimes 211 221 231 241 251 261 271 281 291 301 311 321 331 341 351 361 371 381 391 401 411 421 431 441 451 461 471 481 33 491 Player’s Point of View Occasional freezes Uncomfortable feeling under recommended specs www.gameworks.nvidia.com 34 Stutter is a Huge Thing Focus on our experience with Watch Dogs 2 Catching up issues - Tools Fixing issues - Methods / Approaches www.gameworks.nvidia.com 35 Finding Stutter Cases Watch Dogs 2 is an open world action-adventure game - Each stutter case should be covered by a benchmark www.gameworks.nvidia.com 36 In-game Benchmark We had an internal in-game benchmark, but it didn’t cover all of the cases. www.gameworks.nvidia.com 37 Two Benchmarks In-game Fast Car www.gameworks.nvidia.com 38 Frametime graph (ms) 200 180 160 140 120 100 80 60 40 20 0 Frames www.gameworks.nvidia.com 39 Tools - GPUView - Concurrency visualizer (CV) www.gameworks.nvidia.com 40 Stutter issues 1. GPU Memory consumption GPU View / Afterburner 2. CPU Workload Inconsistency GPU View / CV 3. Bursts of creates / destroys GPU View 4. Page faults CV / GPU View www.gameworks.nvidia.com 41 GPU Memory consumption - Running out of the video memory - Didn’t account some video memory for the driver (~10%) Tools: - GPUView eviction events - GPUView reference charts www.gameworks.nvidia.com 42 GPU Memory Usage (MB) 2700 2500 2300 2100 1900 1700 1500 Frames www.gameworks.nvidia.com 43 UBI – Memory Consumption Approaches •Check VRAM usage (lighting, models, textures, post effects, …) •Give artists the possibility to tune MIP maps skipping on a per texture basis •Look for the allocated but not used resources. •WD2: For the low settings we skipped Global Illumination probes for Far Away objects Challenges •Visual quality on the low settings •Textures in RAM if not enough VRAM www.gameworks.nvidia.com 44 CPU Utilization Inconsistency 100 ms www.gameworks.nvidia.com 45 UBI – CPU Utilization Inconsistency WD2: We use all CPU threads, but leave some idle time for a driver and other processes Frame structure: Main Render Driver Render Render Render Present Worker Engine Render Windows Idle Idle Worker Render Driver Engine Idle Idle Main Engine Engine Engine Engine Idle Idle www.gameworks.nvidia.com 46 UBI – CPU Utilization Inconsistency •Multithread rendering (Using deferred context) •Use Present time to do engine jobs Main Render Present Render DC:Execute DC:Execute Worker Engine Driver Render Driver Worker Engine DC:FillList Render Windows Main Engine Engine Engine Engine DC:FillList www.gameworks.nvidia.com 47 Bursts of creates / destroys Too many of creates / destroys of resources on the render thread www.gameworks.nvidia.com 48 UBI – Resource Management Creating resources on the main render thread: Main Render Resource Resource Resource Render Present Worker Engine Render Engine Idle Idle Worker Render Engine Idle Idle Idle Main Engine Engine Engine Engine Idle Idle www.gameworks.nvidia.com 49 UBI – Resource Management Creating resources on the worker threads: Main Render Render Render Render Render Present Worker Engine Resource Resource Idle Idle Worker Engine Engine Resource Render Idle Main Engine Engine Engine Engine Engine Engine www.gameworks.nvidia.com 50 UBI – Resource Management Resource jobs between the frames: Frame Frame Render Render Render Render Present Render Render Engine Resource Engine Idle Engine Engine Idle Resource Engine Resource Engine Engine Engine Engine Engine Idle Engine Engine Engine www.gameworks.nvidia.com 51 UBI – Resource Management Approaches •Avoid resource creation on the main render thread www.gameworks.nvidia.com 52 Bursts of creates / destroys Too many of creates / destroys of resources on multiple threads www.gameworks.nvidia.com 53 UBI – Bursts of creates / destroys Approaches