3DServices Scalable GPU computing service architecture - GTC-S0261
Hinnerup Net www.hinnerup.net©2012 The LEGO Group Page 1 LEGO 3DServices Scalable GPU computing service architecture GTC-S0261
©2012 The LEGO Group Page 2 Project Background
Customized Content Experiences Game Experiences Building Experiences Improved Online Shopping Experience In-House HPC platform (CPU/GPGPU) In-House Digital Design Software
©2012 The LEGO Group Page 3 Reusing LEGOs 3D technology LXF / LXFML LEGO Model Object
Game 3D Brick BrickKit RenderKit Code Optimizers Assets
LXF / LXFML LEGO Model Object
©2012 The LEGO Group Page 4 Technology Strategy
In-house rendering technologies New digital platform Applications
Technology
Platform
©2012 The LEGO Group Page 5 LEGO 3DServices setup consists of 16 x NVIDIA Quadro Plex 2200 S4 servers
Totalling in:
64 x Quadro FX 5800 GPUs 256 Gb GPU memory 15.360 x CUDA cores
as one expandable/scalable distributed system
©2012 The LEGO Group Page 6 Distributed and Scalable system
DEV: 1 server QA: 1 server STAGING: 2 servers LIVE: 12 servers
4 processes / server
~50.000 CCUs
©2012 The LEGO Group Page 7 Web Based UI
©2012 The LEGO Group Page 8 The Complexity of LEGO Models
©2012 The LEGO Group Page 9 Moderation of User Generated Content
©2012 The LEGO Group Page 10 Moderation of User Generated Content
©2012 The LEGO Group Page 11 Moderation of User Generated Content
~1.650 revisions ~24.000 bricks 8+ Mb LFXML
©2012 The LEGO Group Page 12 Mesh Optimization
©2012 The LEGO Group Page 13 Mesh Optimization
Same LEGO model COP.dll wo. OptiX ~ 1hr 3min 34sec
938 Kb LXFML COP.dll w. OptiX ~ 1min 6sec
©2012 The LEGO Group Page 14 Building Guide Generation
1
2
©2012 The LEGO Group Page 15 Building Guide Generation
1
2
©2012 The LEGO Group Page 16 LEGO 3DServices System Architecture
©2012 The LEGO Group Page 17 System Architecture
Distributed servers
Client application / Web interface HPC on GPU & CPU Calling system / Web services Browser Storage, Web server Application / GUI REST API + CDN DLL (GPU code)
©2012 The LEGO Group Page 18 System Architecture
©2012 The LEGO Group Page 19 System Architecture
IWorker plugin (C#)
Worker Servers Worker processes REST web service
©2012 The LEGO Group Page 20 System Architecture
Database queue UGCC Game servers
REST web service
Cache site Global CDN Gamer / end user / client / system ©2012 The LEGO Group Page 21 System Architecture Technology Platform IWorker plugin Output Providers 3DServices (C#) Worker Service COP-OptiX.dll (C#) (C++ / CUDA & OptiX)
cudart. Optix.1 namespace LEGO.DDDServices.Worker.Entities { dll .dll public interface Iworker { void ProcessModel(Model model); void Close(); Assets } }
©2012 The LEGO Group Page 22 System Architecture
©2012 The LEGO Group Page 23 LEGO 3DServices Development Challenges
©2012 The LEGO Group Page 24 Affinity
GHIC adapter driver if (!wglEnumGpusNV || !wglCreateAffinityDCNV || !wglDeleteDCNV || !wglEnumGpuDevicesNV || !wglEnumGpusFromAffinityDCNV) { errorStrings.PushBack("Affinity not supported by graphics hardware"); return false; } OpenGL Extension for Affinity selection was unavailable on the G-HICx8 frontend card
©2012 The LEGO Group Page 25 Virtual Adapter and Session-0 isolation
Windows Service 4 high-end GPUs OpenGL 1.1 with 2 extensions
Remote Desktop (Hosting / Terremark)
©2012 The LEGO Group Page 26 Automatic Driver Rescue (loop)
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers] "TdrDelay"=dword:0000000a "TdrLevel"=dword:00000000 CUDA Kernel Execution Timeout fix
©2012 The LEGO Group Page 27 GPGPU C/C++ expertise required
©2012 The LEGO Group Page 28 Demo time! (if time permits... ~5 mins.) Credits for the Hero Builder:
Developers : Vladimir Milenov Gorov (app dev) Lars Axberg (web dev) Anders Nygaard (web dev)
Designers: Matias Billeschou (interaction designer) Megan Shellenbarger (GUI designer) Paw Max Andersen (3D models) Martin Lykke (3D models)
©2012 The LEGO Group Page 29 Henrik Høj Madsen [email protected]
Michael Schøler [email protected]
Thanks! Questions?
©2012 The LEGO Group Page 30