3DServices Scalable GPU computing service architecture - GTC-S0261

Hinnerup Net www.hinnerup.net©2012 The LEGO Group Page 1 LEGO 3DServices Scalable GPU computing service architecture GTC-S0261

©2012 The LEGO Group Page 2 Project Background

Customized Content Experiences Game Experiences Building Experiences Improved Online Shopping Experience In-House HPC platform (CPU/GPGPU) In-House Digital Design Software

©2012 The LEGO Group Page 3 Reusing LEGOs 3D technology LXF / LXFML LEGO Model Object

Game 3D Brick BrickKit RenderKit Code Optimizers Assets

LXF / LXFML LEGO Model Object

©2012 The LEGO Group Page 4 Technology Strategy

In-house rendering technologies New digital platform Applications

Technology

Platform

©2012 The LEGO Group Page 5 LEGO 3DServices setup consists of 16 x Plex 2200 S4 servers

Totalling in:

64 x Quadro FX 5800 GPUs 256 Gb GPU memory 15.360 x CUDA cores

as one expandable/scalable distributed system

©2012 The LEGO Group Page 6 Distributed and Scalable system

DEV: 1 server QA: 1 server STAGING: 2 servers LIVE: 12 servers

4 processes / server

~50.000 CCUs

©2012 The LEGO Group Page 7 Web Based UI

©2012 The LEGO Group Page 8 The Complexity of LEGO Models

©2012 The LEGO Group Page 9 Moderation of User Generated Content

©2012 The LEGO Group Page 10 Moderation of User Generated Content

©2012 The LEGO Group Page 11 Moderation of User Generated Content

~1.650 revisions ~24.000 bricks 8+ Mb LFXML

©2012 The LEGO Group Page 12 Mesh Optimization

©2012 The LEGO Group Page 13 Mesh Optimization

Same LEGO model COP.dll wo. OptiX ~ 1hr 3min 34sec

938 Kb LXFML COP.dll w. OptiX ~ 1min 6sec

©2012 The LEGO Group Page 14 Building Guide Generation

1

2

©2012 The LEGO Group Page 15 Building Guide Generation

1

2

©2012 The LEGO Group Page 16 LEGO 3DServices System Architecture

©2012 The LEGO Group Page 17 System Architecture

Distributed servers

Client application / Web interface HPC on GPU & CPU Calling system / Web services Browser Storage, Web server Application / GUI REST API + CDN DLL (GPU code)

©2012 The LEGO Group Page 18 System Architecture

©2012 The LEGO Group Page 19 System Architecture

IWorker plugin (C#)

Worker Servers Worker processes REST web service

©2012 The LEGO Group Page 20 System Architecture

Database queue UGCC Game servers

REST web service

Cache site Global CDN Gamer / end user / client / system ©2012 The LEGO Group Page 21 System Architecture Technology Platform IWorker plugin Output Providers 3DServices (C#) Worker Service COP-OptiX.dll (C#) (C++ / CUDA & OptiX)

cudart. Optix.1 namespace LEGO.DDDServices.Worker.Entities { dll .dll public interface Iworker { void ProcessModel(Model model); void Close(); Assets } }

©2012 The LEGO Group Page 22 System Architecture

©2012 The LEGO Group Page 23 LEGO 3DServices Development Challenges

©2012 The LEGO Group Page 24 Affinity

GHIC adapter driver if (!wglEnumGpusNV || !wglCreateAffinityDCNV || !wglDeleteDCNV || !wglEnumGpuDevicesNV || !wglEnumGpusFromAffinityDCNV) { errorStrings.PushBack("Affinity not supported by graphics hardware"); return false; } OpenGL Extension for Affinity selection was unavailable on the G-HICx8 frontend card

©2012 The LEGO Group Page 25 Virtual Adapter and Session-0 isolation

Windows Service 4 high-end GPUs OpenGL 1.1 with 2 extensions

Remote Desktop (Hosting / Terremark)

©2012 The LEGO Group Page 26 Automatic Driver Rescue (loop)

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers] "TdrDelay"=dword:0000000a "TdrLevel"=dword:00000000 CUDA Kernel Execution Timeout fix

©2012 The LEGO Group Page 27 GPGPU C/C++ expertise required

©2012 The LEGO Group Page 28 Demo time! (if time permits... ~5 mins.) Credits for the Hero Builder:

Developers : Vladimir Milenov Gorov (app dev) Lars Axberg (web dev) Anders Nygaard (web dev)

Designers: Matias Billeschou (interaction designer) Megan Shellenbarger (GUI designer) Paw Max Andersen (3D models) Martin Lykke (3D models)

©2012 The LEGO Group Page 29 Henrik Høj Madsen [email protected]

Michael Schøler [email protected]

Thanks! Questions?

©2012 The LEGO Group Page 30