GPU-Enabled Rapid Image Processing Framework

A GPU-Enabled Rapid Image Processing Framework

Mark Davey Lead HPC Engineer

www.thefoundry.co.uk Visual Effects Software The Foundry

We develop, market and sell Visual Efects Software

NUKE - Compositing MARI - 3D Texture Painting

HIERO - Shot Management

KATANA - Look-Development and Lighting

MODO - 3D Modelling and Rendering www.thefoundry.co.uk Visual Effects Software Aardman Animations BaseFX Studios BlueBolt Who uses our software? Blue Sky Studios Brainstorm Digital CoSA VFX Digital Domain Digital Fusion Double Negative Well established global client base: Dreamworks The Embassy Fido Framestore Igloo VFX Industrial Light and Magic Jellyﬁsh Look Efects Film Lucasﬁlm Entertainment March Entertainment Method Studios Mikros Image The Moving Picture Animation Company Mr. X Passion Pictures Pixomondo Prime Focus Rushes Commercial advertising Smoke and Mirrors Sony Pictures Imageworks Sony Pictures Animation The Mill Township Broadcast Unexpected Union VFX Walt Disney Animation Warner Bros. Animation Weta Digital ZOIC Studios www.thefoundry.co.uk Visual Effects Software At the heart - 2D Image Processing

A fundamental component of our products Used in efects such as: • Noise reduction • Keying • Motion and disparity estimation • Colour correction • 3D texture creation

We need to make it as fast as possible! www.thefoundry.co.uk Visual Effects Software Moving to GPUs

• Traditionally used the CPU for image processing • Lots of legacy code • GPUs are great at image processing • Our customers often have GPUs, but not always (e.g. on render farms) • So need a CPU path • Do not want to write same code multiple times (debugging, maintenance, new hardware, etc) www.thefoundry.co.uk Visual Effects Software Solution: A Rapid Image Processing Framework (RIP)

•Image processing algorithms expressed as kernels •Kernels written in a C++ like, domain-speciﬁc language •Kernels run over an iteration space •Metadata expresses access patterns, image formats, bounday conditions, etc •Kernels are converted to an Abstract Syntax Tree (AST) •The AST is translated into diferent languages

www.thefoundry.co.uk Visual Effects Software Example Kernel

class GainImage : Kernel { param: Image src; Image dst; float gain;

local: void define() { defineParam(gain, "gain", 1.0f); }

void kernel() { dst() = src() * gain; } };

www.thefoundry.co.uk Visual Effects Software Example Kernel

class GainImage : Kernel { param: Image src; Image dst; float gain;

local: void define() { defineParam(gain, "gain", 1.0f); }

void kernel() { dst() = src() * gain; } };

www.thefoundry.co.uk Visual Effects Software Example Kernel

class GainImage : Kernel { param: Image src; Image dst; float gain;

local: void define() { defineParam(gain, "gain", 1.0f); }

void kernel() { dst() = src() * gain; } };

www.thefoundry.co.uk Visual Effects Software Kernel Types

•Iteration Independently run at every point in an iteration space. Example: Gain •Rolling Run over iteration space in order, providing access to results from the previous point along an axis Example: Box blur •Reduction Image data is reduced down to a single ‘value’ example: Maximum pixel value

www.thefoundry.co.uk Visual Effects Software Kernel Metadata

•Granularity - Component | Pixel

•Image access - Read | Write

•Memory Access - Point | Ranged | Random

•Edge Methods - None | Clamped | Constant

www.thefoundry.co.uk Visual Effects Software Current Language Support

CUDA OpenCL C++ Scalar C++ SIMD (SSE2, SSE4.1, AVX)

www.thefoundry.co.uk Visual Effects Software Current RIP-Based Efects

• Denoise • Video Retimer • Convolution • Depth-Based Blur • Motion Blur • Vector Generator

www.thefoundry.co.uk Visual Effects Software Run-time RIP

• Generate kernels at run time (JIT) for specific image formats and data types

• Profile machine to determine best language options to use

• CPU kernels compiled using LLVM

• GPU kernels currently translated to OpenCL www.thefoundry.co.uk Visual Effects Software Example - Denoise

• Proprietary Wavelet-Based Algorithm • Requires 20+ Kernels • Tunable parameters for best results • Must run at interactive speeds • Legacy CPU plug-in too slow

www.thefoundry.co.uk Visual Effects Software Denoise - Tests

GPUs CPUs

Quadro FX 3800M 4 SM Core i7M 2 Core + HT

Quadro K600 1 SMX Core i7-3667U 2 Core + HT

Quadro 4000 8 SM Xeon E5504 4 Core

Quadro 6000 14 SM Xeon X5550 4 Core + HT

Quadro K5000 8 SMX Xeon E5620 8 Core + HT

Geforce GTX 580 16 SM Xeon E5-2640 12 Core + HT

Tesla K20 13 SMX

Image: 1920x1080 (1080p) RGB, 32-bit ﬂoat www.thefoundry.co.uk Visual Effects Software www.thefoundry.co.uk Visual Effects Software www.thefoundry.co.uk Visual Effects Software The RIP Node - Fast R&D

• Develop kernels at run-time within our software using the RIP language • No other development tools required • Automatically creates parameter sliders via kernel introspection • Use graph of nodes to create complex algorithms • Great for rapid research and development www.thefoundry.co.uk Visual Effects Software Example - Directional Blur

CPU: Xeon X5550 GPU: Quadro 6000

Legacy CPU: 227s GPU RIP: 5.6s (40 times faster) GPU RIP Pixel: 3.0s (75 times faster) www.thefoundry.co.uk Visual Effects Software GPU Image Processing - Issues

• Memory is finite and limited • Our software supports very large images • Not always possible to process a whole image on a GPU • Point, and ranged access processing is tiled

• Relatively long transfer times • Try to keep intermediate data on GPU as long as possible • We are working on better caching www.thefoundry.co.uk Visual Effects Software The Future - Highlights

•Beyond 2D image processing • 3D data • Deep data • Arrays of structures •Heterogeneous computing • Use all available devices • Efcient scheduling • Minimise data transfers • Uniﬁed kernel results •Greater GPU optimisation • Better caching • Better use of Kepler architecture