3DServices Scalable GPU computing service architecture - GTC-S0261 Hinnerup Net www.hinnerup.net©2012 The LEGO Group Page 1 LEGO 3DServices Scalable GPU computing service architecture GTC-S0261 ©2012 The LEGO Group Page 2 Project Background Customized Content Experiences Game Experiences Building Experiences Improved Online Shopping Experience In-House HPC platform (CPU/GPGPU) In-House Digital Design Software ©2012 The LEGO Group Page 3 Reusing LEGOs 3D technology LXF / LXFML LEGO Model Object Game 3D Brick BrickKit RenderKit Code Optimizers Assets LXF / LXFML LEGO Model Object ©2012 The LEGO Group Page 4 Technology Strategy In-house rendering technologies New digital platform Applications Technology Platform ©2012 The LEGO Group Page 5 LEGO 3DServices setup consists of 16 x NVIDIA Quadro Plex 2200 S4 servers Totalling in: 64 x Quadro FX 5800 GPUs 256 Gb GPU memory 15.360 x CUDA cores as one expandable/scalable distributed system ©2012 The LEGO Group Page 6 Distributed and Scalable system DEV: 1 server QA: 1 server STAGING: 2 servers LIVE: 12 servers 4 processes / server ~50.000 CCUs ©2012 The LEGO Group Page 7 Web Based UI ©2012 The LEGO Group Page 8 The Complexity of LEGO Models ©2012 The LEGO Group Page 9 Moderation of User Generated Content ©2012 The LEGO Group Page 10 Moderation of User Generated Content ©2012 The LEGO Group Page 11 Moderation of User Generated Content ~1.650 revisions ~24.000 bricks 8+ Mb LFXML ©2012 The LEGO Group Page 12 Mesh Optimization ©2012 The LEGO Group Page 13 Mesh Optimization Same LEGO model COP.dll wo. OptiX ~ 1hr 3min 34sec 938 Kb LXFML COP.dll w. OptiX ~ 1min 6sec ©2012 The LEGO Group Page 14 Building Guide Generation 1 2 ©2012 The LEGO Group Page 15 Building Guide Generation 1 2 ©2012 The LEGO Group Page 16 LEGO 3DServices System Architecture ©2012 The LEGO Group Page 17 System Architecture Distributed servers Client application / Web interface HPC on GPU & CPU Calling system / Web services Browser Storage, Web server Application / GUI REST API + CDN DLL (GPU code) ©2012 The LEGO Group Page 18 System Architecture ©2012 The LEGO Group Page 19 System Architecture IWorker plugin (C#) Worker Servers Worker processes REST web service ©2012 The LEGO Group Page 20 System Architecture Database queue UGCC Game servers REST web service Cache site Global CDN Gamer / end user / client / system ©2012 The LEGO Group Page 21 System Architecture Technology Platform IWorker plugin Output Providers 3DServices (C#) Worker Service COP-OptiX.dll (C#) (C++ / CUDA & OptiX) cudart. Optix.1 namespace LEGO.DDDServices.Worker.Entities { dll .dll public interface Iworker { void ProcessModel(Model model); void Close(); Assets } } ©2012 The LEGO Group Page 22 System Architecture ©2012 The LEGO Group Page 23 LEGO 3DServices Development Challenges ©2012 The LEGO Group Page 24 Affinity GHIC adapter driver if (!wglEnumGpusNV || !wglCreateAffinityDCNV || !wglDeleteDCNV || !wglEnumGpuDevicesNV || !wglEnumGpusFromAffinityDCNV) { errorStrings.PushBack("Affinity not supported by graphics hardware"); return false; } OpenGL Extension for Affinity selection was unavailable on the G-HICx8 frontend card ©2012 The LEGO Group Page 25 Virtual Adapter and Session-0 isolation Windows Service 4 high-end GPUs OpenGL 1.1 with 2 extensions Remote Desktop (Hosting / Terremark) ©2012 The LEGO Group Page 26 Automatic Driver Rescue (loop) Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers] "TdrDelay"=dword:0000000a "TdrLevel"=dword:00000000 CUDA Kernel Execution Timeout fix ©2012 The LEGO Group Page 27 GPGPU C/C++ expertise required ©2012 The LEGO Group Page 28 Demo time! (if time permits... ~5 mins.) Credits for the Hero Builder: Developers : Vladimir Milenov Gorov (app dev) Lars Axberg (web dev) Anders Nygaard (web dev) Designers: Matias Billeschou (interaction designer) Megan Shellenbarger (GUI designer) Paw Max Andersen (3D models) Martin Lykke (3D models) ©2012 The LEGO Group Page 29 Henrik Høj Madsen [email protected] Michael Schøler [email protected] Thanks! Questions? ©2012 The LEGO Group Page 30 .